Visual impairment detection and reinforcement

Abstract

In some embodiments, vision deficiency information may be generated via a dynamic, eye-characteristic-based fixation point. In some embodiments, a first stimulus may be displayed at a first location on a user interface based on the fixation point for the vision test presentation. The fixation point for the vision test presentation may be adjusted during the vision test presentation based on eye characteristic information associated with the user. As an example, the eye characteristic information may indicate a user's eye characteristic that occurred during the vision test presentation. Based on the adjusted fixation point for the vision test presentation, a second stimulus may be displayed at a second interface location on the user interface during the vision test presentation. Vision deficiency information associated with the user may be generated based on feedback information indicating feedback associated with the first stimulus and feedback associated with the second stimulus.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Patent Application No. 16 / 687,512, titled "Vision Defect Determination Via a Dynamic Eye-Characteristic-Based Fixation Point," filed on 18 November 2019. U.S. Patent Application No. 16 / 687,512 is a continuation-in-part application of U.S. Patent Application No. 16 / 654,590, titled "Vision Defect Determination," filed on 16 October 2019. U.S. Patent Application No. 16 / 654,590 claims the interests of U.S. Provisional Application No. 62 / 895,402, filed on September 3, 2019, entitled "Double and Binocular Vision Determination and Correction," and is a continuation-in-part application of U.S. Patent Application No. 16 / 444,604, filed on June 18, 2019, entitled "Vision Defect Determination via a Dynamic Eye-Characteristic-Based Fixation Point." U.S. Patent Application No. 16 / 444,604 is a continuation-in-part application of U.S. Patent Application No. 16 / 428,932, filed on May 31, 2019, entitled "Vision Defect Determination and Enhancement." U.S. Patent Application No. 16 / 428,932 is a continuation of U.S. Patent Application No. 16 / 367,633, filed on 28 March 2019, entitled "Vision Defect Determination and Enhancement Using a Prediction Model." U.S. Patent Application No. 16 / 367,633 is a continuation in part of U.S. Patent Application No. 16 / 144,995, filed on 27 September 2018, entitled "Digital Therapeutic Corrective Spectacles." U.S. Patent Application No. 16 / 144,995 claims the benefits of U.S. Provisional Application No. 62 / 563,770, entitled "Digital Therapeutic Corrective Spectacles," filed on 27 September 2017.Each of these applications is incorporated herein by reference in its entirety. This application also claims priority over the other aforementioned applications filed within at least 12 months of the filing of this application.

[0002] This application is related to i) U.S. Patent Application No. 16 / 662,113, filed on 24 October 2019, entitled "Vision Defect Determination and Enhancement Using a Prediction Model"; ii) U.S. Patent Application No. 16 / 538,057, filed on 12 August 2019, entitled "Vision-Based Alerting Based on Physical Contact Prediction"; iii) U.S. Patent Application No. 16 / 560,212, filed on 4 September 2019, entitled "Field of View Enhancement Via Dynamic Display Portions"; and iv) U.S. Patent Application No. 16 / 428,899, filed on 31 May 2019, entitled "Visual Enhancement for Dynamic Vision Defects". iii) U.S. Patent Application No. 16 / 560,212 is a continuation of U.S. Patent Application No. 16 / 428,380, filed on 31 May 2019, entitled “Field of View Enhancement Via Dynamic Display Portions for a Modified Video Stream.” U.S. Patent Application No. 16 / 428,380 is a continuation of U.S. Patent Application No. 16 / 367,751, entitled “Field of View Enhancement via Dynamic Display Portions,” filed on 28 March 2019. iv) U.S. Patent Application No. 16 / 428,899 is a continuation of U.S. Patent Application No. 16 / 367,687, filed on 28 March 2019, entitled “Visual Enhancement for Dynamic Vision Defects.” Each of these applications is incorporated herein by reference in its entirety. Furthermore, this application claims priority to the aforementioned applications filed at least 12 months prior to the filing of this application.

[0003] This invention relates to facilitating the determination and correction of visual defects related to a user's vision. [Background technology]

[0004] While wearable technologies such as "smart glasses" exist to assist the visually impaired, general wearable technologies do not adequately address many of the problems associated with conventional glasses and contact lenses. For example, general wearable technologies cannot address the problems faced by people with higher-order visual aberrations (such as refractive errors that cannot be corrected with conventional glasses or contact lenses) or dynamic aberrations that change in relation to the eye's accommodation state and gaze direction. The above-mentioned problems and others exist. [Overview of the Initiative]

[0005] Aspects of the present invention relate to methods, apparatus, and / or systems for facilitating the determination and correction of visual defects related to a user's vision. For example, such corrections may include providing enhancement of the user's field of vision or vision (e.g., correction of the user's field of vision or vision, augmentation of the user's field of vision or vision), providing correction of user aberrations, or providing such enhancement or correction via a wearable device.

[0006] In some embodiments, a first stimulus may be displayed at a first position on the user interface based on a fixation point for visual test presentation. The fixation point for visual test presentation may be adjusted during the visual test presentation based on eye characteristic information related to the user. For example, the eye characteristic information may indicate the user's eye characteristics that occurred during the visual test presentation. A second stimulus may be displayed at a second interface position on the user interface during the visual test presentation based on the adjusted fixation point for visual test presentation. Visual defect information associated with the user may be generated based on feedback information indicating feedback related to the first stimulus and feedback related to the second stimulus.

[0007] Various other aspects, features, and advantages of the present invention will become apparent by referring to the detailed description of the invention and the accompanying drawings. Furthermore, both the above summary and the following detailed description are illustrative and should not be understood as limiting the scope of the invention. In this specification and in the claims, the singular forms "a," "an," and "the" imply the plural, unless the context makes it clear that they should be interpreted differently. Furthermore, in this specification and in the claims, the term "or" means "and / or," unless the context makes it clear that they should be interpreted differently. [Brief explanation of the drawing]

[0008] [Figure 1A] One or more embodiments of a system for facilitating user vision-related modifications are shown.

[0009] [Figure 1B] A system for implementing a machine learning model to facilitate user vision-related modifications is shown according to one or more embodiments.

[0010] [Figure 1C] An exemplary eyeglasses device according to one or more embodiments is shown. [Figure 1D] An exemplary eyeglasses device according to one or more embodiments is shown. [Figure 1E] An exemplary eyeglasses device according to one or more embodiments is shown. [Figure 1F] An exemplary eyeglasses device according to one or more embodiments is shown.

[0011] [Figure 2] An exemplary visual system according to one or more embodiments is shown.

[0012] [Figure 3]A diagram showing an apparatus having a vision correction framework implemented in an image processing apparatus and a wearable glasses device according to one or more embodiments.

[0013] [Figure 4] A diagram showing an example of a process including an inspection mode and a vision mode according to one or more embodiments.

[0014] [Figure 5] A diagram showing an example of a process including an inspection mode and a vision mode according to one or more embodiments.

[0015] [Figure 6A] A diagram showing an example of an evaluation protocol for an inspection mode process including pupil tracking according to one or more embodiments. [Figure 6B] A diagram showing an example of an evaluation protocol for an inspection mode process including pupil tracking according to one or more embodiments. [Figure 6C] A diagram showing an example of an evaluation protocol for an inspection mode process including pupil tracking according to one or more embodiments.

[0016] [Figure 7A] A diagram showing an example of an evaluation protocol for an inspection mode process including pupil tracking according to one or more embodiments. [Figure 7B] A diagram showing an example of an evaluation protocol for an inspection mode process including pupil tracking according to one or more embodiments. [Figure 7C] A diagram showing an example of an evaluation protocol for an inspection mode process including pupil tracking according to one or more embodiments.

[0017] [Figure 8] A diagram showing a workflow including an inspection module that generates and presents multiple visual stimuli to a user by a wearable glasses device according to one or more embodiments.

[0018] [Figure 9] The following describes an inspection mode process according to one or more embodiments.

[0019] [Figure 10] This figure shows the process of an artificial intelligence correction algorithm mode that can be implemented as part of an inspection mode, according to one or more embodiments.

[0020] [Figure 11] This figure shows an inspection image according to one or more embodiments.

[0021] [Figure 12] This figure shows how a simulated visual acuity image is generated, which includes overlaying a defective visual field onto an examination image presented to a subject, according to one or more embodiments.

[0022] [Figure 13] This figure shows examples of various modification transformations applied to an image and presented to a subject, according to one or more embodiments.

[0023] [Figure 14] This figure shows examples of translational movement methods according to one or more embodiments.

[0024] [Figure 15] This figure shows examples of machine learning frameworks according to one or more embodiments.

[0025] [Figure 16] This figure shows the process of an AI system of a machine learning framework according to one or more embodiments.

[0026] [Figure 17] This figure shows examples of image transformation according to one or more embodiments.

[0027] [Figure 18] This figure shows examples of translational movement of inspection images according to one or more embodiments.

[0028] [Figure 19] This is a graphical user interface illustrating various implementations of an AI system according to one or more embodiments.

[0029] [Figure 20] This document presents a framework for an AI system including a feedforward neural network, based on one or more embodiments.

[0030] [Figure 21] This figure shows an example of an inspection mode process for an AI system including a neural network, according to one or more embodiments. [Figure 22] This figure shows an example of an inspection mode process for an AI system, including an AI algorithm optimization process, according to one or more embodiments.

[0031] [Figure 23] This figure shows an example of a process for performing inspection mode and visual mode according to one or more embodiments.

[0032] [Figure 24A] The present invention describes a wearable eyewear device including custom reality wearable glasses that, according to one or more embodiments, allow images from the environment to pass through a transparent portion of the display of the wearable glasses, where the transparent portion corresponds to the peripheral area of ​​the user's field of view, and the other portion of the display of the wearable glasses is opaque.

[0033] [Figure 24B]The present invention describes a wearable eyewear device including custom reality wearable glasses that, according to one or more embodiments, allow images from the environment to pass through a transparent portion of the display of the wearable glasses, where the transparent portion corresponds to the central area of ​​the user's field of view, and the other portion of the display of the wearable glasses is opaque.

[0034] [Figure 24C] According to one or more embodiments, eye tracking is used to demonstrate alignment between a field of view plane, a remapped image plane, and a selective transparent screen plane.

[0035] [Figure 25A] This document describes use cases in which a visual examination presentation is displayed to a patient without strabismus according to one or more embodiments.

[0036] [Figure 25B] This document describes use cases in which a visual examination presentation is displayed to a patient with strabismus according to one or more embodiments.

[0037] [Figure 25C] This document describes the automatic measurement and correction of diplopia according to one or more embodiments. [Figure 25D] This document describes the automatic measurement and correction of diplopia according to one or more embodiments. [Figure 25E] This document describes the automatic measurement and correction of diplopia according to one or more embodiments. [Figure 25F] This document describes the automatic measurement and correction of diplopia according to one or more embodiments. [Figure 25G] This document describes the automatic measurement and correction of diplopia according to one or more embodiments. [Figure 25H] This document describes the automatic measurement and correction of diplopia according to one or more embodiments. [Figure 25I] This document describes the automatic measurement and correction of diplopia according to one or more embodiments.

[0038] [Figure 25J] This figure shows a binocular vision test and its results according to one or more embodiments. [Figure 25K] This figure shows a binocular vision test and its results according to one or more embodiments. [Figure 25L] This figure shows a binocular vision test and its results according to one or more embodiments.

[0039] [Figure 25M] This shows a stereoscopic vision test according to one or more embodiments. [Figure 25N] This shows a stereoscopic vision test according to one or more embodiments.

[0040] [Figure 26] This diagram illustrates the normal binocular vision of a subject, showing how a single perceptual image is generated by combining a monocular image of the left eye and a monocular image of the right eye, including the central macula and the peripheral visual field surrounding that central region.

[0041] [Figure 27] This diagram shows visual field constriction, a condition in which the peripheral area is not visible to the subject.

[0042] [Figure 28] This figure shows an image shift technique for correcting visual field constriction or improving visual acuity, according to one or more embodiments.

[0043] [Figure 29] This figure illustrates an image resizing transformation technique for improving visual acuity or maintaining central visual acuity while expanding the field of view, according to one or more embodiments.

[0044] [Figure 30] This figure shows a binocular field of view expansion technology according to one or more embodiments.

[0045] [Figure 31A] This figure illustrates a technique for evaluating dry eye and corneal abnormalities, which includes projecting a pattern onto the corneal surface and imaging the corneal surface that reflects the pattern, according to one or more embodiments.

[0046] [Figure 31B] This figure schematically illustrates the presentation of a reference image, including a grid pattern, displayed to or projected onto the subject's cornea or retina, by wearable glasses according to one or more embodiments.

[0047] [Figure 31C] Examples of grid patterns for subject manipulation according to one or more embodiments are shown.

[0048] [Figure 31D] This figure shows an example of manipulating the grid pattern shown in Figure 31C according to one or more embodiments.

[0049] [Figure 31E] This figure shows a scene to be perceived by a subject, according to one or more embodiments.

[0050] [Figure 31F] This figure shows an example of a corrected visual field, which a subject would perceive when provided to a subject with visual distortion determined by a grid pattern technique according to one or more embodiments, as illustrated in Figure 31E.

[0051] [Figure 31G] This figure shows a display, according to one or more embodiments, that includes an operable grid pattern for a subject to communicate distortion within their field of view.

[0052] [Figure 32] This figure shows an image of the corneal surface reflecting a pattern projected onto the corneal surface, according to one or more embodiments.

[0053] [Figure 33] This figure shows an example of normal pattern reflection according to one or more embodiments.

[0054] [Figure 34] This figure shows examples of abnormal pattern reflections according to one or more embodiments.

[0055] [Figure 35A] This figure shows a presentation of a visual examination using a dynamic fixation point, according to one or more embodiments. [Figure 35B] This figure shows a presentation of a visual examination using a dynamic fixation point, according to one or more embodiments. [Figure 35C] This figure shows a presentation of a visual examination using a dynamic fixation point, according to one or more embodiments. [Figure 35D] This figure shows a presentation of a visual examination using a dynamic fixation point, according to one or more embodiments. [Figure 35E] This figure shows a presentation of a visual examination using a dynamic fixation point, according to one or more embodiments.

[0056] [Figure 35F] This figure shows a flowchart relating to a process for facilitating the presentation of a visual examination using a dynamic fixation point, according to one or more embodiments.

[0057] [Figure 35G] This describes the presentation of a visual test including multiple contrast-staircase stimuli and stimulus sequences at a predetermined location, according to one or more embodiments.

[0058] [Figure 36] This is a timing diagram showing the processing of a test sequence at a single stimulus location according to one or more embodiments.

[0059] [Figure 37]This figure illustrates the calculation of the width and height of pixels that define the boundary of the maximum brightfield, according to one or more embodiments.

[0060] [Figure 38] This figure shows an examination image used to examine the four main quadrants of the visual field, according to one or more embodiments.

[0061] [Figure 39A] This figure shows examples of the field of view before remapping, according to one or more embodiments.

[0062] [Figure 39B] This figure shows examples of fields of view after remapping according to one or more embodiments.

[0063] [Figure 40A] This figure shows examples of custom reality glasses devices according to one or more embodiments. [Figure 40B] This figure shows examples of custom reality glasses devices according to one or more embodiments. [Figure 40C] This figure shows examples of custom reality glasses devices according to one or more embodiments.

[0064] [Figure 41] A flowchart illustrating a method for facilitating user visual correction via a predictive model, according to one or more embodiments, is provided.

[0065] [Figure 42] A flowchart illustrating a method for facilitating the expansion of a user's field of view through a combination of parts of multiple images of a scene, according to one or more embodiments, is provided.

[0066] [Figure 43]A flowchart illustrating a method for facilitating enhancement of a user's field of view through one or more dynamic display portions on one or more transparent displays, according to one or more embodiments, is shown. [Modes for carrying out the invention]

[0067] In the following description, many specific and detailed details are included for illustrative purposes so that the embodiments of the present invention can be fully understood. However, those skilled in the art will understand that embodiments of the present invention can be carried out without these specific and detailed details, or with equivalent configurations. In other cases, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring each embodiment of the present invention.

[0068] Figure 1A shows a system 100 for facilitating user visual modifications according to one or more embodiments. As shown in Figure 1, the system 100 may comprise a server 102, client devices 104 (or client devices 104a-104n) or other components. The server 102 may include a configuration subsystem 112, a model manager subsystem 114 or other components. The client devices 104 may include an inspection subsystem 122, a visual subsystem 124 or other components. Each of the client devices 104 may include any type of mobile terminal, fixed terminal, or other device. For example, a client device 104 may include a desktop computer, a notebook computer, a tablet computer, a smartphone, a wearable device, or other client device. The user may use, for example, one or more client devices 104 to interact with each other, with one or more servers, or with other components of the system 100.

[0069] While one or more processes described herein as being performed by specific components of the client device 104, it should be noted that in some embodiments, these processes may be performed by other components of the client device 104 or other components of the system 100. For example, one or more processes described herein as being performed by components of the client device 104 may, in some embodiments, be performed by components of the server 102. Similarly, while one or more processes described herein as being performed by specific components of the server 102, it should be noted that in some embodiments, these processes may be performed by other components of the server 102 or other components of the system 100. For example, one or more processes described herein as being performed by components of the server 102 may, in some embodiments, be performed by components of the client device 104. Furthermore, while several embodiments of machine learning models are described herein, it should be noted that in other embodiments, other predictive models (e.g., statistical models or other analytical models) may be used instead of or in addition to the machine learning models (for example, in one or more embodiments, a statistical model may replace the machine learning model, and a non-statistical model may replace the non-machine learning model).

[0070] In some embodiments, the system 100 may provide the user with a presentation of a visual test, which includes a set of stimuli (e.g., light stimuli, text, or images displayed to the user). During (or after) the presentation, the system 100 may obtain feedback related to the set of stimuli (e.g., feedback indicating whether or how the user is looking at one or more of the stimuli in the set). As an example, the feedback may include an indication of the user's response to one or more stimuli (of the set), or an indication of the user's lack of response to such stimuli. The response (or lack thereof) may relate to eye movements, gaze direction, changes in pupil size, or user modifications or other user inputs (e.g., the user's response or other response to a stimulus). As another example, the feedback may include an image of the eye taken during the presentation of the visual test. The eye image may be an image of the retina of the eye (e.g., the whole or a portion of the retina), an image of the cornea of ​​the eye (e.g., the whole or a portion of the cornea), or other eye images.

[0071] In some embodiments, system 100 may determine one or more defective areas of the user's field of view (e.g., an automatic determination based on feedback related to a set of stimuli displayed to the user or other feedback). For example, a defective area may be one of the areas of the user's field of view that fails to meet one or more visual criteria (e.g., whether or not the user perceives one or more stimuli, or the degree to which they perceive them, light sensitivity, distortion, or other aberrations, or other criteria). In some embodiments, system 100 may provide an enhanced image or adjust one or more configurations of a wearable device based on the determination of the defective area. For example, an enhanced image may be generated or displayed to the user such that one or more predetermined portions thereof (e.g., a region of the enhanced image corresponding to the macular region of the user's eye's field of view or a region within the macular region of the eye) are outside the defective area. In another example, the position, shape, or size of one or more display portions of a wearable device, the brightness, contrast, saturation, or sharpness level of the display portion, the transparency of the display portion, or other configurations of the wearable device may be adjusted based on the determined defective area.

[0072] In some embodiments, one or more predictive models may be used to facilitate the determination of visual defects (e.g., light sensitivity, distortion, or other aberrations), the determination of correction profiles (e.g., correction / enhancement profiles including correction parameters or functions) used to correct or enhance the user's vision, the generation of enhanced images (e.g., obtained from live image data), or other operations. In some embodiments, the predictive models may include one or more neural networks or other machine learning models. As an example, the neural network may be based on a large number of neural units (or artificial neurons). The neural network may roughly mimic the workings of a biological brain (e.g., due to large clusters of biological neurons connected by axons). Each of the neural units in the neural network may be coupled to many other neural units in the neural network. Such couplings may have a reinforcing or inhibitory effect on the activity state of the connected neural units. In some embodiments, each neural unit may have an aggregation function that combines all input values. In some embodiments, each connection (or the neural unit itself) may have a threshold function, where the signal must exceed the threshold before it propagates to other neural units. These neural network systems can learn and train automatically rather than be explicitly programmed, and may perform significantly better than conventional computer programs in solving problems in a given domain. In some embodiments, the neural network may have multiple layers (for example, the signal path traverses from higher to lower layers). In some embodiments, the neural network may utilize backpropagation, in which forward stimuli are used to reset the weights of neural units in the "higher layers". In some embodiments, the stimuli and inhibitions to the neural network may be more fluid as the interactions of the connections become more chaotic and complex.

[0073] As an example, as shown in Figure 1B, the machine learning model 162 may take input 164 and provide output 166. In one use case, output 166 may be fed back to the machine learning model 162 as input for training the machine learning model 162 (e.g., alone or in combination with user instructions regarding the accuracy of output 166, labels associated with the input, or other reference feedback information). In another use case, the machine learning model 162 may update its configuration (e.g., weights, biases, or other parameters) based on an evaluation of the prediction (e.g., output 166) and the reference feedback information (e.g., user instructions regarding accuracy, reference labels, or other information). In yet another use case, if the machine learning model 162 is a neural network, the connected weights may be adjusted to adjust for the difference between the neural network's prediction and the reference feedback. In a further use case, one or more neurons (or nodes) in the neural network may request that their respective errors be sent backward through the neural network to facilitate an update process (e.g., backpropagation). The update of the connected weights may reflect, for example, the magnitude of the error to be propagated backward after forward propagation is complete. In this way, for example, the predictive model may be trained to generate better predictions.

[0074] In some embodiments, once the system 100 obtains feedback relating to a set of stimuli (displayed to the user), feedback relating to one or more of the user's eyes, feedback relating to the user's environment, or other feedback, it may provide this feedback to a predictive model, which may be configured based on the feedback. For example, the predictive model may be automatically configured for the user based on (i) a display of the user's response to one or more stimuli (of the set of stimuli), (ii) a display of the user's lack of response to such stimuli, (iii) an eye image captured during the presentation of the visual test, or other feedback (for example, the predictive model may be personalized for the user based on feedback from the presentation of the visual test). In another example, the predictive model may be trained based on such feedback and other feedback from other users to improve the accuracy of the results provided by the predictive model. In some embodiments, once the predictive model is configured (for example, for the user), the system 100 may provide live image data or other data to the predictive model to obtain and display an augmented image (obtained from the live image data). As an example, a wearable device of system 100 may acquire a live video stream from one or more cameras of the wearable device and display an enhanced image on one or more displays of the wearable device. In some embodiments, the wearable device may acquire an enhanced image (e.g., a file or other data structure representing the enhanced image) from a predictive model. In some embodiments, the wearable device may acquire a correction profile (e.g., correction parameters or functions) from the predictive model and generate an enhanced image based on the live video stream and the correction profile. In one use case, the correction profile may include correction parameters or functions used to generate an enhanced image from live image data (e.g., parameters of a function used to convert or correct live image data into an enhanced image).Furthermore, or alternatively, the modification profile may include modification parameters or functions for dynamically configuring one or more display portions (e.g., dynamic adjustment of transparent or opaque portions of a transparent display, dynamic adjustment of the projection portion of a projector).

[0075] In some embodiments, the system 100 can facilitate the enhancement of the user's field of view through one or more dynamic display parts (e.g., the transparent display part of a transparent display, the projection part of a projector, etc.). As an example, with respect to a transparent display, the dynamic display part may include one or more transparent display parts and one or more other display parts (e.g., the display parts of a wearable device or other device). The system 100 may cause one or more images to be displayed on the other display parts. As an example, the user can see through the transparent display part of the transparent display but not the other display parts, and instead can see the image presentation on the other display parts of the transparent display (e.g., parts around or adjacent to the transparent display part). In one use case, live image data may be acquired via a wearable device, and an enhanced image may be generated based on the live image data and displayed on the other display parts of the wearable device. In some embodiments, the system 100 may monitor one or more changes related to one or more of the user's eyes and, based on the monitoring, adjust the transparent display part of the transparent display. As an example, the changes to be monitored may include eye movements, changes in gaze direction, changes in pupil size, or other changes. The position, shape, size, transparency, or other aspects of one or more transparent display portions of a wearable device may be automatically adjusted based on monitored changes. In this way, for example, system 100 may improve mobility without restricting (or at least reducing) eye movements, gaze direction, pupillary response, or other changes related to the eyeball.

[0076] In some embodiments, the system 100 may facilitate the expansion of the user's field of view through a combination of parts of multiple images of a scene (for example, based on feedback related to a set of stimuli displayed to the user or other feedback), and the system 100 may acquire multiple images of a scene. The system 100 may determine a region common to the multiple images, and for each of the multiple images, it may determine a region of the image that is different from a corresponding region of at least one other image among the multiple images. In some embodiments, the system 100 may generate or display an enhanced image to the user based on the common and different regions. As an example, the common and different regions may be combined to generate an enhanced image that includes representations of the common and different regions. The common region may correspond to each part of multiple images having the same or similar characteristics to each other, and each different region may correspond to a part of one image that is different from all the corresponding parts of the other images. In a scenario, the part of one image that is different from the others may include a part of the scene that is not represented in the other images. In this way, for example, by combining the common and different regions to create an enhanced image, the field of view that would otherwise be provided for each image is expanded, and the user's vision can be enhanced using the enhanced image.

[0077] In some embodiments, the system 100 may generate a prediction that an object will physically come into contact with the user and display an alert based on this physical contact prediction (e.g., an alert related to the object is displayed on the user's wearable device). In some embodiments, the system 100 may detect an object in a defective portion of the user's field of view and display an alert based on (i) that the object is in a defective portion of the field of view, (ii) a physical contact prediction, or (iii) other information. In some embodiments, the system 100 may determine whether an object is outside (or not fully within) any portion of the enhanced image (displayed to the user) corresponding to at least one portion of the field of view that satisfies one or more visual criteria. In one use case, if it is determined that an object is within (or fully within) a portion of the enhanced image corresponding to a healthy portion of the user's field of view, no alert may be displayed (or a lower-priority alert may be displayed) (e.g., even if the object is predicted to physically come into contact with the user). On the other hand, if an object within a defective field of view is predicted to make physical contact with the user, and it is determined that the object is outside (or not sufficiently within) the user's healthy field of view, an alert may be displayed on the user's wearable device. In this way, for example, the user can rely on their own healthy field of view to avoid objects approaching within their healthy field of view, thereby reducing the risk of relying on the wearable device (e.g., through habit formation) to avoid such approaching objects. However, it should be noted that in other use cases, an alert related to the object may be displayed based on the prediction of physical contact, regardless of whether the object is within the user's healthy field of view.

[0078] In some embodiments, with respect to Figure 1C, the client device 104 may include an eyeglasses device 170 that forms a wearable device for the subject. In some embodiments, the eyeglasses device 170 may be part of the visual system described herein. The eyeglasses device 170 comprises a left eyepiece lens 172 and a right eyepiece lens 174. The eyepiece lenses 172 and 174 may each include a digital monitor configured to display a regenerated image to the corresponding eye of the subject (e.g., projected onto a screen or onto the eye), or may be associated with such a digital monitor. In various embodiments, the digital monitor may include a display screen, a projector, and / or hardware that generates an image to be displayed on the display screen or projects the image onto the eye (e.g., the retina of the eye). It should be conceivable that the digital monitor, including the projector, may be positioned elsewhere to project the image onto the subject's eye or onto the eyepiece lens, including a screen, eyeglasses, or other image projection surface. In one embodiment, the left and right axiolytic lenses 172 and 174 may be positioned relative to the housing 176 to conform to the orbital region of the subject, enabling data collection and display / projection of image data. This may include, in another example, displaying / projecting image data to a different eye.

[0079] Each of the eye lenses 172, 174 may further have one or more inward-facing sensors 178, 180, which may be inward-facing image sensors. In one example, the inward-facing sensors 178, 180 may include an infrared camera, photodetector, or other infrared sensor configured to track pupil movement and determine and track the subject's visual axis. The inward-facing sensors 178, 180 (including, for example, an infrared camera) may be positioned lower than the eye lenses 172, 174. This ensures that the subject's field of view, whether the actual field of view or the field of view displayed or projected to the subject, is not obstructed. The inward-facing sensors 178, 180 may be oriented toward the assumed pupil region to better track the pupil and / or line of sight. In some examples, the inward-facing sensors 178, 180 may be embedded in the eye lenses 172, 174 so as to form a continuous inner surface.

[0080] Figure 1D is a front view of the eyeglasses device 170, showing the eyepieces 172 and 174 viewed from the front. Corresponding outward-facing image sensors 182 and 184, which constitute a field-of-view camera, are positioned on the eyepieces 172 and 174. In other embodiments, the number of outward-facing image sensors 182 and 184 provided may be increased or decreased. The outward-facing image sensors 182 and 184 may be configured to capture multiple consecutive images. The eyeglasses device 170 or the corresponding visual system may further be configured to subsequently correct and / or enhance the images. This may be done on an individual basis based on the subject's visual impairment. The eyeglasses device 170 may further be configured to display the corrected and / or enhanced images to the subject using a monitor in visual mode. For example, the eyeglasses device may generate the corrected and / or enhanced images on a display screen associated with the eyepieces or adjacent areas, project the images onto a display screen associated with the eyepieces or adjacent areas, or project the images onto one or both of the subject's eyes.

[0081] Figures 1E and 1F show other examples of the eyeglasses device 170. With respect to Figures 1E and 1F, the eyeglasses device 170 includes one or more high-resolution cameras 192, a power supply unit 193, a processing unit 194, a glass screen 195, a see-through display 196 (e.g., a transparent display), an eye tracking system 197, and other components.

[0082] In some embodiments, the eyeglasses device 170 may have an examination mode. In one example of an examination mode, the inward-facing sensors 178, 180 track pupil movement and perform tracking of the visual axis (e.g., line of sight) according to the examination protocol. In this example or another, the inward-facing sensors 178, 180 may be configured to image reflections of patterns reflected by the cornea and / or retina in order to detect distortions and abnormalities of the corneal or ocular optics.

[0083] The examination mode may be used to perform a visual assessment to identify eye diseases, such as higher-order and / or lower-order aberrations, optic nerve diseases such as glaucoma, optic neuritis and optic neuropathy, retinal diseases such as macular degeneration and retinitis pigmentosa, visual pathway diseases such as capillary disorders and tumors, and other conditions such as presbyopia, strabismus, higher-order and lower-order aberrations, monocular vision, anisometropia and aniseikonia, photosensitivity, pupillary inequality, refractive errors and astigmatism. In the examination mode, data is collected for a specific subject and the images acquired using this data are corrected before displaying these images. Display of the images may include projecting them onto the subject via a monitor, as described herein.

[0084] In some cases, external sensors may be used to provide additional data for evaluating the subject's visual field. For example, data used to correct captured images may be obtained from external testing devices such as visual field analyzers, aberration meters, electrooculographs, or visual evoked potential devices. The data obtained from these devices can be combined with pupil or eye-tracking to determine the visual axis to create one or more correction profiles (e.g., correction profiles, enhancement profiles, etc., used to correct or enhance such images) used to correct the image projected or displayed to the user.

[0085] The eyeglasses device 170 may have a visual mode in addition to, or instead of, the inspection mode. In the visual mode, one or more outward-facing image sensors 182, 184 capture images that are transmitted to an image processor for real-time image processing. The image processor may be embedded in the eyeglasses device 10, for example. Alternatively, it may be located outside the eyeglasses device 170, for example, and may be associated with an external image processing device. The image processor may be a component of the visual module and / or may include a scene processing module, as described herein.

[0086] The eyeglasses device 170 may be communicably coupled to one or more image processors by wired or wireless communication, for example, via a wireless transceiver embedded in the eyeglasses device 170. The external image processor may include a computer such as a laptop computer, tablet, mobile phone, network server, or other centralized or distributed computer processing unit, and may be characterized by one or more processors and one or more memories. According to the described example, the captured image is processed by this external image processing unit. However, in other examples, the captured image may be processed by an image processor embedded in the digital eyeglasses. The processed image (e.g., enhanced to improve visual acuity in terms of functional field or other aspects, and / or enhanced to correct a visual field disorder in the subject) is then transmitted to the eyeglasses device 170 and displayed on a monitor for the subject to view.

[0087] In one example of the operation of a visual system including eyeglasses, real-time image processing of captured images may be performed by an image processor (for example, using customized MATLAB code) on a small computer embedded in the eyeglasses. In other examples, the code may be performed by an external image processing device or by another computer connected to a wireless network to communicate with the eyeglasses. In one embodiment, the visual system includes eyeglasses, an image processor, and corresponding instructions for executing a visual mode and / or an examination mode, such instructions may be implemented in the eyeglasses alone or in combination with one or more external devices (such as a laptop computer). Such a visual system may operate in two modes: a visual mode and a separate examination mode.

[0088] In some embodiments, as shown in Figure 2, the system 100 may include a vision system 200 comprising an eyeglasses device 202 which is communicably coupled to a network 204 for communication with a server 206, a mobile phone 208, or a personal computer 210. The server 206, mobile phone 208, or personal computer 210 may each include a vision correction framework 212 that implements processing techniques described herein, such as image processing techniques. This may include techniques relating to examination modes and / or vision modes. In the illustrated example, the vision correction framework 212 has a processor, a memory that stores an operating system and applications for implementing the techniques described herein, and further has a transceiver for communicating with the eyeglasses device 202 via the network 204. The framework 212 includes an examination module 214 which includes a machine learning framework in this example. The machine learning framework may be used with an examination protocol executed by the examination module, either in a supervised or unsupervised manner, to adaptively adjust the examination mode to more accurately assess eye diseases. The results of processing by the testing module may include the generation of a customized visual correction model 216 for the subject 218.

[0089] The visual module 220 generates corrected visual images to be displayed by the eyeglasses device 202, which in some embodiments may also include a machine learning framework with an accessed and customized visual correction model. The visual correction framework 212 may further include a scene processing module that processes images used in the processing of the inspection mode and / or visual mode, and includes processing that is described herein in relation to the processing module. As described herein, in some embodiments, the eyeglasses device 202 may include some or all of the visual correction framework 212.

[0090] In the examination mode, one or more inward-facing image sensors, including a tracking camera positioned along the inner surface of the spectacle device 170 or 202, may be used to acquire pupil and visual axis tracking data, which is used to precisely align the processed image with the pupil and visual axis of the subject.

[0091] In some embodiments, as shown in Figure 3, the system 100 may include a visual system 300, which includes a visual correction framework 302. The visual correction framework 302 may be implemented in an image processing device 304 and in eyeglasses 306 worn by a subject. The image processing device 304 may be entirely contained within an external image processing device or other computer, and in other examples, part or all of the image processing device 304 may be implemented within the eyeglasses 306.

[0092] The image processing device 304 may have a memory 308 that stores instructions 310 for performing the inspection mode and / or visual mode described herein. Such instructions may include instructions for collecting high-resolution images of the subject from the glasses device 306. In the visual mode, the glasses device 306 may acquire real-time field of view image data as raw data, processed data, or pre-processed data. In the inspection mode, the glasses device may project an inspection image (such as the word "text" or an image of a vehicle or other object) to examine the subject's field of view at various points.

[0093] The eyeglasses device 306 may be communicably connected to the image processing device 304 via a wired or wireless link. This link may be via a Universal Serial Bus (USB), IEEE 1394 (FireWire), Ethernet, or other wired communication protocol device. Wireless connectivity can be provided via any suitable wireless communication protocol such as WiFi, NFC, iBeacon, Bluetooth, or Bluetooth Low Energy.

[0094] In various embodiments, the image processing apparatus 304 may have a controller operably connected to a database via a link connected to input / output (I / O) circuits. An additional database may be connected to this controller in a known manner. This controller may include program memory, a processor (also called a microcontroller or microprocessor), random access memory (RAM), and input / output (I / O) circuits, all interconnected via an address / data bus. While only one microprocessor is described, it should be noted that the controller may include multiple microprocessors. Similarly, the controller's memory may include multiple RAMs and multiple program memories. The RAM(s) and program memories may be implemented as semiconductor memory, magnetically readable memory, and / or optically readable memory. The link may operably connect the controller to a capture device via the I / O circuits.

[0095] The program memory and / or RAM may store various applications (i.e., machine-readable instructions) executed by the microprocessor. For example, the operating system may control the overall operation of the visual system 300, for example, the operation of the eyeglasses device 306 and / or the image processing device 304, and in some embodiments, it may provide a user interface to the device to implement the operations described herein. The program memory and / or RAM may further store various subroutines for accessing specific functions of the image processing device 304 described herein. As an example, but not limited to, subroutines may include, in particular, acquiring high-resolution images of the field of view from the eyeglasses device, enhancing and / or correcting the images, and providing the enhanced and / or corrected images for display to the subject by the eyeglasses device 306.

[0096] In addition to the above, the image processing apparatus 304 may include other hardware resources. The apparatus may further include various types of input / output hardware, such as a visual display and one or more input devices (e.g., a keypad, keyboard, etc.). In one embodiment, the display may be touch-sensitive and may accept user input in cooperation with a software keyboard routine as one of the software routines. It may be beneficial for the image processing apparatus 304 to communicate with a broader network (not shown) using one of many known network devices and network technologies (e.g., via a computer network such as an intranet or the Internet). For example, the apparatus may be connected to a database of aberration data.

[0097] In some embodiments, the system 100 may store predictive models, correction profiles, visual impairment information (e.g., indicating a detected user's visual impairment), feedback information (e.g., feedback related to stimuli displayed to the user, or other feedback), or other information in one or more remote databases (e.g., the cloud). In some embodiments, feedback information, visual impairment information, correction profiles, or other information related to multiple users (e.g., two or more users, ten or more users, one hundred or more users, one thousand or more users, one million or more users, or any other number of users) may be used to train one or more predictive models. In some embodiments, one or more predictive models may be trained or configured for a user or a predetermined type of device (e.g., a device of a specific brand, a device of a specific brand and model, a device with a specific set of features, etc.) and may be stored in association with the user or device type. As an example, instances of a predictive model associated with a user or device type may be stored locally (e.g., on the user's wearable device or other user devices) and remotely (e.g., in the cloud), and such instances of the predictive model may be automatically or manually synchronized between one or more user devices and the cloud so that the user can access the latest configuration of the predictive model on either the user device or the cloud. In some embodiments, multiple modification profiles may be associated with a user or device type. In some embodiments, each modification profile may include a set of modification parameters or functions applied to live image data in a given context to generate an enhanced presentation of the live image data. As an example, a user may have modification profiles for each set of eye characteristics (e.g., range of gaze direction, pupil size, margin position, or other characteristics). As a further example, a user may additionally or alternatively have modification profiles for each set of environmental characteristics (e.g., range of ambient light levels, ambient temperature, or other characteristics).Based on the currently detected eye or environmental characteristics, a corresponding set of modification parameters or functions may be obtained and used to generate an enhanced presentation of live image data.

[0098] Subsystems 112-124

[0099] In some embodiments, with respect to Figure 1A, the examination subsystem 122 may provide the user with a visual examination presentation. As an example, the presentation may include a set of stimuli. During (or after) the presentation, the examination subsystem 122 may obtain feedback related to the set of stimuli (e.g., feedback indicating whether or how the user is looking at one or more stimuli from the set). As an example, the feedback may include an indication of the user's response to one or more stimuli (from the set), or an indication of the user's lack of response to such stimuli. The response (or lack thereof) may relate to eye movements, gaze direction, changes in pupil size, or user modifications or other user inputs (e.g., the user's response or other response to a stimulus). As another example, the feedback may include an image of the eye taken during the presentation of the visual examination. The eye image may be an image of the retina of the eye (e.g., the whole or a portion of the retina), an image of the cornea of ​​the eye (e.g., the whole or a portion of the cornea), or other eye images. In some embodiments, the inspection subsystem 122 may, based on the feedback, generate one or more results, such as the affected portion of the user's field of vision, the extent of the affected portion, the user's visual pathology, a correction profile for correcting the aforementioned problem, or other results.

[0100] In some embodiments, based on feedback related to a set of stimuli (displayed to the user during a visual test presentation) or other feedback, the testing subsystem 122 may determine light sensitivity, distortion, or other aberrations related to one or more of the user's eyes. In some embodiments, the set of stimuli may include a pattern, and the testing subsystem 122 may project the pattern onto one or more of the user's eyes (e.g., using a projection-based wearable eyeglasses device). For example, the pattern can be projected onto the user's retina or cornea to determine defects affecting the retina or cornea. In one use case, the projected pattern can be used to correct and evaluate dyspmorphopsia in age-related macular degeneration and other retinal diseases. As shown in Figure 31A, a digital projection of the pattern 3100 may be projected onto the subject's eye 3102. The pattern may be digitally generated by a projector placed inside the eyeglasses device. A digital camera 3104 (e.g., an inward-facing image sensor) may also be placed inside the eyeglasses device to capture an image of the pattern 3100 reflected by the eye 3102. For example, the image may be captured from the corneal surface of the eye, as shown in Figure 32. After capturing an image of pattern 3100, the examination subsystem 122 may determine whether the pattern looks normal (for example, as shown in Figure 33) or abnormal (for example, as shown in Figure 34(3101)). Abnormalities may be evaluated and corrected using one of the techniques described herein.

[0101] In some embodiments, the inspection subsystem 122 may display a set of stimuli to the user, acquire images of one or more of the user's eyes (e.g., at least a portion of the user's retina or cornea) as feedback related to the set of stimuli, and determine one or more correction parameters or functions to address light sensitivity, distortion, or other aberrations (e.g., low-order or high-order aberrations, static or dynamic aberrations) related to the user's eyes. Such corrections may include transformations (e.g., rotation, reflection, translation / shift, resizing), adjustments to image parameters (e.g., brightness, contrast, saturation, sharpness), or other modifications. As an example, when a pattern (e.g., an Amsler grid or other pattern) is projected onto the user's retina or cornea, the resulting image may include reflections of the aberrated projected pattern (e.g., those reflected from the retina or cornea). The inspection subsystem 122 may automatically determine correction parameters or functions to apply to the pattern such that, when the corrected pattern is projected onto the retina or cornea, the (subsequently obtained) image of the retina or cornea is one or more aberration-free versions of the uncorrected pattern image. In one use case, with respect to Figure 31C, when pattern 3100 is projected onto the user's retina, the resulting image may include a distorted version of pattern 3100 (for example, an inverse version of the distortion depicted in the modified pattern 3100' in Figure 31D). The parameters of a function such as (for example, one that reverses the distortion of the obtained image) may be determined and applied to pattern 3100 to generate a modified pattern 3100'. Modified pattern 3 When 100' is projected onto the user's retina, the corrected pattern 3100' from the user's retina The reflection will include the uncorrected, undistorted pattern 3100 in Figure 31C. To the extent that the reflection still contains distortion, the inspection subsystem 122 may automatically update the correction parameters or functions applied to the pattern to further mitigate the distortion (e.g., as shown in the retinal reflection).

[0102] In another use case, an image of the eye (e.g., an image of one or more of the user's eyes) capturing a projected stimulus (e.g., a pattern or other stimulus) reflected from the retina or cornea may be used to determine a function (or parameters of a function) for correcting one or more other aberrations. The determined function or parameters may be applied to the projected stimulus, and as long as the reflection of the corrected stimulus still contains aberrations, the inspection subsystem 122 may automatically update the corrected parameters or functions applied to the stimulus to further reduce the aberrations (e.g., those shown in the reflection). In a further use case, the aforementioned automatic determination of parameters or functions may be performed for each of the user's eyes. In this way, for example, appropriate parameters or functions for each eye may be used to provide correction for anisometropia or other conditions in which each eye has different aberrations. For example, with respect to anisometropia, conventional corrective glasses cannot correct the unequal refractive power of the two eyes. This is because corrective glasses create two images of unequal size (e.g., one for each eye) (aniseikonia), and the brain cannot merge these two images for binocular monopsia, resulting in visual confusion. The cause of this problem is simple: eyeglass lenses are either convex, which magnifies the image, or concave, which reduces the image. The degree of magnification or reduction varies depending on the amount of correction. Assuming that appropriate parameters or functions can be determined for each eye, the aforementioned operation (or other techniques described herein) can correct anisometropia (along with other symptoms in which each eye has different aberrations), thereby avoiding visual confusion or other problems associated with such symptoms.

[0103] In some embodiments, with respect to Figure 1A, the examination subsystem 122 can display a set of stimuli to the user and determine one or more modification parameters or functions (addressing light sensitivity, distortion, or other aberrations related to the user's eye) based on the user's modifications to the set of stimuli or other user input. In some scenarios, with respect to Figure 31C, the pattern 3100 may be a grid diagram (e.g., an Amsler grid) or any known reference shape designed to detect transformations necessary to treat one or more ocular abnormalities. The image may then be distorted in reverse in real time using these transformations to improve vision. In the implementation example of Figure 8, the visual system 800 may include an examination module 802. The examination module 802 may be associated with wearable glasses or may be operated in combination with an external device as described herein. The examination module 802 may present examination stimuli, including an Amsler grid, to the subject 806. The subject may improve the distortion by manipulating the image of the grid diagram via the user device 808 or other input device (for example, by dragging or moving one or more parts of the lines of the grid diagram). The visual correction framework 810 may present the Amsler grid for further correction by the subject. Once the subject completes the manual correction (which, for example, creates a corrected pattern 3100'), the visual correction framework 810 may generate a correction profile of the subject to be applied to the scene as seen when using the spectacle device. As an example, the visual correction framework 810 may output an inverse function (or the same) that outputs a corrected pattern 3100' when pattern 3100 is provided as input to the function. Parameters for such a function may be generated. The workflow described for the visual system 800 may be similarly applicable to processing other inspection modes described herein.

[0104] Figure 31B is a schematic diagram illustrating how the Amsler grid 3100 (e.g., an example of a reference image) is presented as an image on wearable glasses (e.g., a VR headset or AR headset). The Amsler grid 3100 may be displayed or projected onto the subject's cornea and / or retina. An example of a standard grid diagram 3100 is shown in Figure 31C. The same grid pattern may be displayed on the user device. The subject may manipulate the lines of the grid pattern, particularly the lines that appear curved, using a keyboard, mouse, touchscreen, or other inputs on the user device, including a user interface. The subject can identify a reference point 3102 to begin image manipulation. After identifying the reference point, the subject may adjust the identified lines using the user device (e.g., arrow keys) to correct the distortion perceived due to an abnormal macula. Since this procedure is performed separately for each eye, two corrected grid diagrams may be obtained.

[0105] Once the subject completes the line modification so that it appears as a straight line, the visual correction framework uses the new grid diagram to generate a mesh of intersections corresponding to the added distortion. Such a mesh generated in the inspection mode is applied to any image to compensate for the subject's anomaly. For example, as part of the verification of the inspection mode, the modified image corresponding to the appropriate mesh may be shown to each eye. The subject can then use the user device to point out whether any defects are found in the corrected image, and if no defects are found, the correction is considered successful. For example, Figure 31E illustrates the actual scene that the user should perceive. Figure 31F shows the corrected field of view, and if provided to a subject with visual distortion determined by the Amsler grid method, a subject viewing the field of view in Figure 31F will perceive the actual field of view in Figure 31E.

[0106] Such corrections may be performed in real time on the live image, and the corrected visual scene may be continuously presented to the subject. The correction may be achieved in real time because a correction mesh may be available, whether or not the eyeglasses device includes a display that generates the imaging field of view, or whether or not the eyeglasses device is a custom reality type and utilizes a correction layer to adjust for distortion.

[0107] In some cases, reference images such as the Amsler pattern may be displayed directly on a touchscreen or tablet PC, such as the 3150 (e.g., a tablet PC) shown in Figure 31G. The Amsler pattern is presented on the display of the device 3150, and the subject may use a stylus 3152 to manipulate the lines that appear curved to draw corrections to be applied to the lines so that they appear straight. During the examination mode, the grid diagram may be redrawn each time a change is made to reflect the latest edits. This procedure is performed separately for each eye, so two corrected grid diagrams may be obtained. After the subject completes the changes in the examination mode, the tablet PC runs an application to create mesh data and send it to an attached application on the eyeglass device, and processes the image to which the determined mesh is applied.

[0108] The spectacle device, upon receiving the results of the changes in the examination mode, may apply them to any image to compensate for any abnormalities in the subject. It may then display the image obtained through this correction. The display may be done via a VR headset or an AR headset. For example, the display may use the headset to present the image to the user in a holographic manner. Each of the displayed images may correspond to a mesh generated for each eye. If the corrected image appears to the subject free of abnormalities, the correction may be considered successful and may be retained for future image processing. In some embodiments of the examination mode, instead of presenting a single image modified according to the modified grid diagram, or in addition, a video incorporating the changes may be presented. In one example, the video includes a live video stream from a camera supplied after the correction and is displayed to the subject.

[0109] In some embodiments, with respect to Figure 1A, the inspection subsystem 122 may determine one or more defective areas of the user's field of view (e.g., automatic determination based on feedback related to a set of stimuli displayed to the user or other feedback). For example, a defective area may be one of the areas of the user's field of view that fails to meet one or more visual criteria (e.g., whether or not the user perceives one or more stimuli, the degree of perception, light sensitivity, distortion, or other aberrations, or other criteria). In some cases, the set of stimuli displayed to the user includes at least one inspection image of text or objects. A defective area may include areas with reduced visual sensitivity, areas with high or low optical aberrations, areas with reduced brightness, or other defective areas of the field of view. In some cases, the sets of stimuli may differ in contrast levels from each other by at least 20 dB, and further from a reference contrast level. In some cases, the sets of stimuli may differ in contrast levels from each other by at least 30 dB, and further from a reference contrast level. In some cases, the testing subsystem 122 can instruct the wearable glasses device to display a set of testing stimuli to the user in descending or ascending order of contrast during testing.

[0110] In one use case, the test was conducted on four subjects. The test protocol involved displaying a string of characters in multiple different positions on one or more display monitors of the spectacle device. To assess the blind spots in the subjects' visual fields, the word "text" was displayed on each eye's spectacle monitor, and the subjects were asked if they could identify "text". First, the operator intentionally placed the "xt" part of the word "text" in the subjects' blind spots. All four subjects reported that they could only see the "te" part of the word. Next, the characters were moved using software to control the display. The string of characters "text" was moved away from the subjects' blind spots, and the subjects were asked to read the word again. The subjects were able to read "text" and reported that they could now see the "xt" part of the word.

[0111] An example of this evaluation protocol for the test mode is shown in Figures 6A to 6C. As illustrated in Figures 6A and 6B, the code automatically detects blind spots in the Humphrey visual field. The word "text" 600 is projected so that the "xt" portion of the word is located at the blind spot 602 (Figure 6A). The subject is asked to read the word. The word "text" 600 is then moved away from the blind spot 602 (Figure 6B), and the subject is asked to read it again. The word "text" 600 can be displayed at multiple different coordinates in the subject's visual field, and in the illustrated example, the visual field is divided into four coordinates. This protocol makes it possible to identify multiple blind spots, including the peripheral blind spot 604. The string may be moved across the subject's entire visual field, and the subject may be asked to identify if part or all of the string is not visible, or if it is partially visible or visible but with reduced intensity.

[0112] The pupil tracking function described herein may include the physical state of the pupil (e.g., visual axis, pupil size and / or corneal margin), alignment, dilation and / or line of sight. The line of sight, also known as the visual axis, may be obtained by tracking one or more of the pupil, the corneal margin (the margin between the cornea and the sclera), or by tracking blood vessels on the surface of the eye or inside the eye. Thus, pupil tracking may also include tracking of the corneal margin or blood vessels. Pupil tracking may be performed using one or more inward-facing image sensors, as described herein. In various embodiments, the pupil tracking function may be used to determine parameters for aligning the projected image to the subject's field of view (Figure 6C).

[0113] With respect to Figure 6C, the GUI606 display may be shown to the operator. The GUI606 may provide information related to the examination. For example, the GUI606 may show the measured visual field defect and the relative position of the image to the defect. The GUI606 may be configured to automatically distribute images to the functional parts of the visual field, but may include a button to allow the operator to override the automatic mode. An external image processing device may be configured to determine where this evaluation string should be displayed and may wirelessly instruct the digital glasses to display the string at various positions in the examination mode.

[0114] In another use case, as shown in Figures 7A to 7C, instead of using "text," subjects were tested to determine whether they could see vehicle 700 positioned in multiple different parts of their field of vision in order to track their pupils and determine the lesion area. The pupil tracking function allows the visual system to align the projected image with the subject's field of vision.

[0115] In some embodiments, with respect to Figure 1A, the examination subsystem 122 may determine one or more defective portions of the user's visual field based on the user's eye response to a set of stimuli presented to the user, or the absence of a response to a set of stimuli (a response being, for example, an eye movement response, a pupil size response, etc.). In some embodiments, one or more stimuli may be dynamically presented to the user as part of a visual examination presentation, and the response to the stimuli or the absence of a response may be recorded and used to determine which portions of the user's visual field are healthy. For example, if the user's eyes respond to a presented stimulus (e.g., change the direction of their gaze toward the location of the presented stimulus), the eye response may be used to indicate that the eyes can see the presented stimulus (e.g., the corresponding portion of the user's visual field is part of the user's healthy visual field). On the other hand, if the user's eyes do not respond to a presented stimulus (e.g., do not change the direction of their gaze toward the location of the presented stimulus), the absence of an eye response may be used to indicate that the eyes cannot see the presented stimulus (e.g., the corresponding portion of the user's visual field is a defective portion of the visual field). Based on the instructions above, the inspection subsystem 122 may automatically determine the defective portion of the user's field of view.

[0116] In some embodiments, the set of stimuli displayed to the user may include stimuli of different brightness, contrast, saturation, or sharpness levels, and a response to or lack thereof to a stimulus having a particular brightness, contrast, saturation, or sharpness level may indicate whether a portion of the user's field of vision (the portion corresponding to the location of the displayed stimulus) has a problem related to brightness, contrast, saturation, or sharpness. For example, if the user's eyes respond to a displayed stimulus having a particular brightness level, the eye response may be used to indicate that the eyes can see the displayed stimulus (e.g., that the corresponding portion of the user's field of vision is part of the user's healthy field of vision). On the other hand, if the user's eyes do not respond to a stimulus at the same location but with a lower brightness level (e.g., one that a normal eye would respond to), the lack of eye response can be used to indicate that the brightness of the corresponding portion of the user's field of vision is reduced. In some cases, the brightness level of the stimulus may be gradually increased until the user's eyes respond to the stimulus or until a certain brightness level threshold is reached. If the user's eyes finally respond to the stimulus, the current brightness level may be used to determine the level of light sensitivity of the corresponding portion of the field of vision. If the user's eyes do not respond to a stimulus even when a brightness level threshold is reached, the corresponding portion of the visual field may be determined to be a blind spot (for example, if the eyes do not respond when one or more stimuli corresponding to contrast, saturation, sharpness, etc., are changed). Based on the above instructions, the inspection subsystem 122 may automatically determine the defective portion of the user's visual field.

[0117] In some embodiments, the fixation point for visual test presentation may be dynamically determined. In some embodiments, the position of the fixation point and the position of the stimulus displayed to the user may be dynamically determined based on the user's line of sight or other aspects. For example, during the presentation of a visual test, both the fixation point and the position of the stimulus are dynamically presented to the patient relative to the patient's eye movements. In one use case, the current fixation point may be set at the position of the visual test presentation that the patient is currently looking at a particular instance of, and the test stimulus may be displayed relative to that fixation point. In this way, for example, the patient does not need to fixate their attention on a predetermined specific fixation point. This can make the presentation of the visual test more objective and interactive and reduce the stress caused by fixing on a fixed point for a long time. Also, using a dynamic fixation point can eliminate patient errors related to fixation (for example, if the patient forgets to focus on a static fixation point).

[0118] In some embodiments, a fixation point may be locked, and one or more test stimuli may be displayed relative to that fixation point until the lock is released (e.g., Figure 35F). Once released, the current fixation point may be set to the position of the visual test presentation in which the patient is currently viewing a particular instance. A new fixation point may then be locked, and one or more subsequent test stimuli may be displayed relative to that new fixation point. In some embodiments, while the fixation point remains the same, multiple stimuli may be displayed at one or more different positions in the visual test presentation. For example, while the fixation point remains the same, one or more stimuli may be displayed, then de-highlighted / de-highlighted, then one or more other stimuli may be displayed, then de-highlighted / de-highlighted. In one use case, to de-emphasize the stimulus, the brightness or other intensity level of the stimulus may be reduced (e.g., reduced by a predetermined amount, reduced to a default "low" threshold level, reduced to a patient-specific threshold level determined to be invisible to the patient, etc.). In another use case, the stimulus may be removed from the user interface (e.g., the stimulus is no longer displayed by the user interface).

[0119] As described above, in some embodiments, the examination subsystem 122 may adjust the fixation point (e.g., for the visual examination presentation) based on eye characteristic information related to the user (e.g., the patient's eye movements, gaze direction, or other eye-related characteristics that occur during the visual examination presentation). In one use case, the examination subsystem 122 may display a first stimulus at a first interface position on a user interface (e.g., the user's wearable device or other device) based on the fixation point. The examination subsystem 122 may adjust the fixation point based on eye characteristic information and, based on the adjusted fixation point, display a second stimulus at a second interface position on the user interface during the visual examination presentation. As described above, in some embodiments, one or more stimuli may be displayed on the user interface (e.g., at different interface positions) between the display of the first stimulus and the display of the second stimulus. The examination subsystem 122 may acquire feedback information during the visual examination presentation and generate visual defect information based on such feedback information. As an example, the feedback information may indicate feedback related to a first stimulus, feedback related to a second stimulus, feedback related to a third stimulus displayed during the visual test presentation, or feedback related to one or more other stimuli. Such feedback may indicate (i) the user's response to a stimulus, (ii) the user's lack of response to a stimulus, (iii) whether or not the user perceives one or more stimuli, or the degree of perception, light sensitivity, distortion, or other aberrations, or (iv) other feedback. The generated visual defect information can be used to (i) train one or more predictive models, (ii) determine one or more corrective profiles of the user, (iii) facilitate live image processing to correct or modify images for the user, or (iv) perform other actions described herein.

[0120] In some embodiments, using a dynamic fixation point during the presentation of a visual inspection allows for wider coverage of the user's field of view than the dimensions of the view provided through the user interface. For example, as shown with respect to Figures 35A-35E, the user interface (e.g., the user's wearable device or other device) may be configured to display a view having one or more dimensions, where each dimension corresponds to a degree (e.g., a width of 70 degrees, a height of 70 degrees, a width or height of another degree, etc.). However, by using a dynamic fixation point, the inspection subsystem 122 may generate visual defect information with coverage exceeding the degrees of one or more dimensions (e.g., the horizontal dimension of the user's field of view compared to the width of the user interface view, the vertical dimension of the user's field of view compared to the height of the user interface view, etc.). In one scenario, based on these techniques, the visual defect information may have coverage of an area up to 2.85 times larger than the entire user interface view, and the coverage area may be expanded to a size approaching four times the size of the entire user interface view (for example, if the distance between the wearable device and the user's eyes decreases, or the distance between the two monitors of the wearable device increases). Furthermore, the visual defect information may have coverage of up to twice the width of the user's field of view than the width of the user interface view, up to twice the height of the user's field of view than the height of the user interface view, or other expanded areas of the user's field of view. In another scenario, based on these techniques, the visual defect information may indicate whether or to what extent defects exist at two or more field of view locations in the user's field of view, where the field of view locations are farther apart from each other with respect to one or more dimensions of the user's field of view than the degree of the dimensions of the user interface view.

[0121] In one use case, with respect to Figure 35A, the use of a dynamic fixation point and user interface 3502 (which is configured, for example, to provide a 70-degree view) can facilitate the generation of a field map 3504 having coverage greater than 70 degrees in both horizontal and vertical dimensions. As an example, as shown in Figure 35A, the stimulus 3506a may be displayed at the center of the user interface 3502, causing the user to look at the center of the user interface 3502, thereby initializing the fixation point 3508 at the center of the user interface 3502. Specifically, if characteristics of the user's eye (e.g., characteristics detected by the eye-tracking technique described herein) indicate that the user is looking at stimulus 3506a, the fixation point 3508 for visual examination presentation may be set to the position on the user interface 3502 currently corresponding to stimulus 3506a. In some use cases, the fixation point "floats" on the user interface 3502 depending on where the user is currently looking.

[0122] In another use case, as shown in Figure 35B, stimulus 3506b may be displayed in the lower left corner of the user interface 3502 (for example, 50 degrees away from the position of the user interface 3502 where stimulus 3506a is displayed). If the user's eye characteristics indicate that the user is sensing stimulus 3506b (for example, if the user's eye movements are detected as being directed toward stimulus 3506b), the visual field map 3504 may be updated to indicate that the user can see the corresponding position in the user's visual field (for example, 50 degrees away in the same direction from the fixation point position in the visual field map 3504). If the user's eye characteristics indicate that the user is currently looking at stimulus 3506b, the fixation point 3508 for the visual test presentation may then be set to the position of the user interface corresponding to stimulus 3506b.

[0123] In another use case, as shown in Figure 35C, stimulus 3506c may be displayed in the upper right corner of the user interface 3502 (for example, 100 degrees away from the position of the user interface 3502 where stimulus 3506b is displayed). If the user's eye characteristics indicate that the user is perceiving stimulus 3506c, the visual field map 3504 may be updated to indicate that the user can see the corresponding position in the user's visual field (for example, 100 degrees away in the same direction from the fixation point position on the visual field map 3504). If the user's eye characteristics indicate that the user is currently looking at stimulus 3506c, the fixation point 3508 for the visual test presentation may then be set to the position of the user interface 3502 corresponding to stimulus 3506c. As shown in Figure 35D, stimulus 3506d may be displayed in the lower left corner of the user interface 3502 (for example, 100 degrees away from the position of the user interface 3502 where stimulus 3506c is displayed). If the user's ophthalmic characteristics indicate that the user is sensing stimulus 3506d, the visual field map 3504 may be updated to indicate that the user can see the corresponding location in the user's visual field (for example, a location 100 degrees away in the same direction from the fixation point location on the visual field map 3504). If the user's ophthalmic characteristics indicate that the user is currently looking at stimulus 3506d, the fixation point 3508 for the visual test presentation may then be set to the location on the user interface 3502 corresponding to stimulus 3506d. As shown in Figure 35E, stimulus 3506e may be displayed to the left of the upper right corner of the user interface 3502 (for example, a location 90 degrees away from the location on the user interface 3502 where stimulus 3506d is displayed). If the user's ophthalmic characteristics indicate that the user is sensing stimulus 3506e, the visual field map 3504 may be updated to indicate that the user can see the corresponding location in the user's visual field (for example, a location 90 degrees away in the same direction from the fixation point location on the visual field map 3504).If the characteristics of the user's eyes indicate that the user is currently looking at stimulus 3506b, the fixation point 3508 for the visual examination presentation may then be set to the position of the user interface 3502 corresponding to stimulus 3506e. In this way, for example, even if the user interface view is only 70 degrees in both the horizontal and vertical dimensions, the visual field map 3504 currently has coverage corresponding to 200 degrees oblique of the user's visual field, 140 degrees of the user's visual field in the horizontal dimension, and 140 degrees of the user's visual field in the vertical dimension.

[0124] In another use case, with respect to Figure 35B, if the characteristics of the user's eyes indicate that the user is not looking at stimulus 3506b (e.g., there was no significant eye movement response to the display of stimulus 3506b, or the user's gaze did not move to an area adjacent to the location of stimulus 3506b on the user interface 3502), the field map 3504 may be updated to indicate that the user is unable to see the corresponding location in the user's field of view. Thus, in some scenarios, the field map may indicate visual defects in the user's field of view and their corresponding locations for areas larger than the size of the view of the user interface 3502. For example, even if the user interface view is only 70 degrees in the horizontal and vertical dimensions, the field map may indicate visual defects at field locations that are 70 degrees or more apart from each other in the horizontal and vertical dimensions, respectively (e.g., the distance between such indicated visual defects may be up to 140 degrees in the horizontal dimension and up to 140 degrees in the vertical dimension).

[0125] In some embodiments, to facilitate wider coverage of the user's field of view (for example, despite limitations of hardware / software components related to the user interface view), one or more positions on the user interface may be selected to display one or more stimuli based on interface positions further from the current fixation point (for example, for visual test presentations). In some embodiments, the test subsystem 122 may select a first interface position on the user interface based on the fact that the first interface position is further from the fixation point than one or more other interface positions on the user interface, and display a first stimulus at the first interface position. In some embodiments, after the fixation point has been adjusted (for example, based on characteristics relating to the user's eyes), the test subsystem 122 may select a second interface position on the user interface based on the fact that the second interface position is further from the adjusted fixation point than one or more other interface positions on the user interface, and display a second stimulus at the second interface position.

[0126] As an example, a first stimulus may be selected to be added to a queue of stimuli to be displayed during a visual test presentation (e.g., a queue of stimuli to be displayed next) based on (i) the first stimulus being associated with a first visual field location in the user's visual field, and (ii) the first visual field location corresponding to a first interface location (e.g., determined by the fixation point and the position of the first visual field location relative to the fixation point). As a further example, a second stimulus may be selected to be added to a queue during a visual test presentation based on (i) the second stimulus being associated with a second visual field location in the user's visual field, and (ii) the second visual field location corresponding to a second interface location. By selecting a “further” stimulus / location to be displayed next, the test subsystem 122 adjusts the fixation point to a location farther away from the center of the user interface view, thereby increasing the coverage of the user's visual field. In one use case, with respect to Figure 35B, stimulus 3506b and its corresponding position on the user interface 3502 are selected as the next stimulus / position to be displayed during the visual test presentation, based on the determination that the position of the corresponding interface is one of the furthest positions on the user interface from the fixation point (located at the center of the user interface 3502). In this way, the fixation point is adjusted to the lower left corner of the user interface 3502 (for example, by having the user look there), thereby allowing the next stimulus to be displayed as far as 100 degrees away from the fixation point (for example, the distance between stimulus 3506b and stimulus 3506c in Figure 35C).

[0127] In some embodiments, one or more locations in the user's visual field may be included as part of a set of visual field locations to be tested during a visual test presentation. For example, the set of visual field locations to be tested may be represented by stimuli during the visual test presentation, and the determination of whether or not the user has a visual defect, or the degree thereof, at one or more visual field locations in the test set is based on whether or not the user perceives one or more of the corresponding stimuli, or the degree thereof. In some embodiments, a visual field location may be removed from the test set based on the determination that the visual field location has been sufficiently tested (for example, by displaying a stimuli at the corresponding location on a user interface and detecting whether or not the user perceives the displayed stimuli, or the degree thereof). For example, the removal of a visual field location may include labeling the visual field location in the test set as no longer being selectable from the test set during the visual test presentation. Thus, in some scenarios, the stimuli corresponding to the removed visual field location may not be displayed during subsequent visual test presentations, while stimuli corresponding to one or more other visual field locations in the test set may be displayed during subsequent visual test presentations. In further scenarios, visual field locations may subsequently be added to the test set (for example, by labeling the visual field locations in the test set as selectable during the visual test presentation, or by removing any prior labels indicating that the visual field locations are not selectable during the visual test presentation).

[0128] In some embodiments, if the fixation point is adjusted to a first user interface position on the user interface where a first stimulus is displayed during a visual test presentation, the test subsystem 122 may display one or more stimuli on the user interface based on the fixation point at the first interface position. The test subsystem 122 may also subsequently display a second stimulus at a second interface position on the user interface. For example, the second stimulus may be displayed while the fixation point is still at the first interface position (for example, the fixation point may be locked at the first interface position until just before the second stimulus is displayed, until the second stimulus is displayed, or until any other time). In some embodiments, the test subsystem 122 may detect that the user's eyes have fixed on the second interface position based on eye characteristic information related to the user, and the test subsystem 122 may adjust the fixation point to the second interface position based on the fixation detection.

[0129] In some embodiments, the testing subsystem 122 may prevent adjustment (or readjustment) of the fixation point to a different interface position on the user interface while the lock is established by establishing a lock on the fixation point for visual test presentation. In this way, for example, one or more stimuli can be displayed on the user interface while the fixation point lock is established to test one or more positions in the user's field of view relative to the locked fixation point. After the lock on the fixation point is released, the fixation point may be dynamically adjusted again. As an example, the testing subsystem 122 may have a stimulus presented at a new interface position on the user interface (different from the interface position where the fixation point was set). Based on the detection that the user's eyes have fixed on the new interface position, and after the lock on the fixation point has been released, the testing subsystem 122 may adjust the fixation point to the new interface position. In one use case, as described above with respect to Figure 35F, the fixation point lock may be released so that the user can "capture" the stimulus (e.g., step 3544), and then, based on the user looking at the stimulus, the fixation point lock may be restored at the new interface position (e.g., step 3546). Specifically, when a user "locks onto" a stimulus (and is still looking at it), the location of this stimulus becomes a new fixation point.

[0130] In some embodiments, while a fixation point is located at a first interface location on the user interface, the inspection subsystem 122 may display multiple stimuli at interface locations different from the first interface location. For example, one or more of the multiple stimuli may be displayed on the user interface, followed by the display of one or more other stimuli. As another example, after a stimulus is displayed on the user interface, it may be dehighlighted or removed from the user interface, and then after another stimulus is subsequently displayed on the user interface, it may be dehighlighted or removed from the user interface. In one use case, with respect to Figure 35F, the fixation point may be locked at the interface location (where the preceding stimulus was displayed) (e.g., step 3546), and based on the fixation point, one or more stimuli may be displayed on the user interface at new interface locations (e.g., steps 3528, 3540a, etc.).

[0131] In another use case, multiple locations in the user's field of view may be examined by displaying multiple stimuli at different interface locations while the fixation point remains locked. For example, with respect to Figure 35C, the fixation point may alternatively be locked to the interface location where stimulus 3506b is displayed in the user interface 3502, and the portion of the user's field of view corresponding to the upper right corner of the field of view map 3504 may be examined by displaying stimuli at different locations in the user interface 3502 while the fixation point remains locked to the interface location of stimulus 3506b.

[0132] In some embodiments, one or more interface locations on the user interface may be pre-designated to be relative fixation points on which the user's visual field is examined. For example, if the four corners of the user interface are pre-designated to be fixation points during the visual examination presentation, the examination subsystem 122 may initially display a stimulus in the center of the user interface so that the user first fixates on the central stimulus (e.g., the initial fixation point). The examination subsystem 122 may then display a stimulus in the upper right corner of the user interface and, upon detecting that the user is looking at the upper right stimulus (e.g., based on the characteristics of the user's eyes), adjust and lock the fixation point to the upper right corner of the user interface. The examination subsystem 122 may then examine a portion of the user's visual field by displaying stimuli at different locations on the user interface while the fixation point remains locked. In one use case, if the user interface is represented by the user interface 3502 in Figure 35A and the user's field of view is represented by the field of view map 3504 in Figure 35A, the portion of the user's field of view corresponding to the lower left quarter of the field of view map 3504 may be thoroughly tested by displaying stimuli at different positions on the user interface while the fixation point remains locked to the upper right corner. The above process may then be repeated for other corners of the user interface to examine the portion of the user's field of view corresponding to other parts of the field of view map 3504.

[0133] In some embodiments, while the fixation point is at a first interface location on the user interface (where the first stimulus is displayed), the inspection subsystem 122 may display multiple stimuli, then de-highlight or remove them from the user interface, while the first stimulus remains displayed at the first interface location on the user interface. For example, if the first interface location is the upper right corner of the user interface, the first stimulus may remain displayed while a series of other stimuli are momentarily displayed on the user interface. Thus, a visual change occurring at another interface location (due to another stimulus appearing at that interface location) causes the user to see the source of the visual change if the other interface location does not correspond to a defective area of ​​the user's field of vision (e.g., a blind spot). However, once the other stimuli disappear, the first stimulus becomes the primary (or sole) source of the visual simulation for the user's eye, and the user returns to fixating on the upper right corner.

[0134] In some embodiments, while the fixation point is at a first interface position on the user interface (where the first stimulus is displayed), the inspection subsystem 122 may de-emphasize or remove the first stimulus from the user interface, and then emphasize or re-display it at the first interface position on the user interface. In some embodiments, while the fixation point is at the first interface position, the inspection subsystem 122 may display multiple stimuli on the user interface, and after the display of at least one of the multiple stimuli, emphasize or re-display the first stimulus at the first interface position on the user interface. In one use case, if the brightness of the first stimulus decreases, the brightness of the first stimulus may be increased so that the user's eyes perceive a visual change (and increased visual stimulus) and return to fixate on the first interface position where the first stimulus is displayed on the user interface. In another use case, if the first stimulus is removed from the user interface, re-displaying the first stimulus similarly causes the user's eyes to return to fixate on the first interface position on the user interface.

[0135] In some embodiments, one or more parts of the process shown in Figure 35F may be used to facilitate the presentation of a visual examination using a dynamic fixation point. With respect to Figure 35F, step 3522 is a matrix of possible stimuli (e.g., all possible stimuli) in the user's visual field. Step 3524 is an eye-tracking device used to lock a floating fixation point to the center of the visual field. For example, the eye coordinates obtained from the eye-tracking device may be used to "float" the floating fixation point around the eye. Step 3526 is a ranking of the available stimuli in the matrix (e.g., the point furthest from the floating fixation point comes first). For example, a stimuli corresponding to a user interface view location that is at least the same distance from the fixation point as all other locations on the user interface (corresponding to the available stimuli in the matrix) may be ranked higher than all other available stimuli (or ranked with the same priority as other stimuli that are equally far from the floating fixation point). For example, the ranking may be performed in real time using an eye-tracking device (e.g., pupil or eye-tracking device or other eye-tracking device).

[0136] In step 3528, following the ranking, the first stimulus on the ranking list (e.g., the stimulus with the highest priority) may be the next stimulus to be displayed during the visual test presentation. For example, the stimulus may be displayed in a color that provides high contrast to the background (e.g., the stimulus may be black to provide contrast to a black background). In step 3530, the eye movement vector (or other representation of an eye-related characteristic) may be consistently measured using an eye-tracking device. If it is detected that the eye movement is not directed toward the stimulus (step 3532), in step 3534, the stimulus is counted as not seen and removed from the matrix of available stimuli. Steps 3528–3530 are repeated using the current highest-ranked stimulus on the ranking list (which is in the matrix of available stimuli).

[0137] If eye movement toward a stimulus is detected (step 3536) (for example, thereby indicating that the user is perceiving the stimulus), in step 3538 the stimulus is counted as (qualitatively) seen, and the stimulus disappears from the user interface. In steps 3540a-3540d, a visual test presentation may be used to test the extent to which the user can perceive the stimulus in a specific area of ​​their visual field. For example, in step 3540a, the stimulus is displayed again, but with each step performed the color shade becomes darker (e.g., a shade of gray). In one use case, the stimulus may be displayed again initially in a color similar to the background color (e.g., if the background color is white, the stimulus color may initially be light gray). In step 3540b, the eye movement vector (or other expression of an eye-related characteristic) may be continuously measured using an eye-tracking device. If it is detected that the eye movement is not directed toward the stimulus (step 3540c), steps 3540a and 3540b are repeated (for example, using a darker shade of color to create more contrast with the white background). If it is detected that the eye movement is directed toward the stimulus, the visual sensitivity for a specific area of ​​the visual field is indicated based on the degree of shading of the displayed stimulus color (for example, the degree of gray shading) (step 3542).

[0138] In step 3544, the eye-tracking / floating fixation point is unlocked (for example, to allow the user to capture a stimulus). In step 3546, the eye-tracking / floating fixation point is restored (for example, based on where the user is currently looking). For example, if the user has "captured" a stimulus (and is still looking at it), the location of this stimulus becomes the new floating fixation point. In step 3548, this stimulus is removed from the matrix of available stimuli, and the process in step 3526 is repeated for the other available stimuli in the matrix.

[0139] In some embodiments, the fixation point location or the location of the stimulus displayed to the user may be static during the visual test presentation. For example, the test subsystem 122 may display a stimulus at the center of the user interface (or at a location corresponding to a static fixation point) to cause the user to look at the center of the user interface (or at another location corresponding to a static fixation point). When it is detected that the user is looking at the static fixation point location, the test subsystem 122 may display the next stimulus from a set of stimuli for testing one or more areas of the user's visual field. Each time it is detected that the user is not looking at the static fixation point location, the test subsystem 122 may repeatedly display a stimulus at the static fixation point location.

[0140] As another example, with respect to Figure 35G, a visual test presentation applying a fast thresholding strategy may utilize a sequence of 52 stimuli at a given location, using four contrasting stepped stimuli covering a 40-degree radius around the center. Other examples may use a different number of contrasting stimuli, coverage, and stimulus positions. In this example, stimuli were placed at the center of each cell illustrated in Figure 35G. Twelve corner cells were excluded from the test because the stimuli were not visible due to the circular lens of the display. The spacing between stimulus positions was approximately 10 degrees. Each stimulus sequence contained four consecutive stimuli with different contrast levels relative to the background. The contrast of the stimuli varied between 33 dB and 24 dB, in descending increments of 3 dB between each contrast level. The threshold was recorded with the last stimulus shown. If the patient did not see any contrast of any stimulus at a particular location, this location was considered invisible and given a value of 0 dB.

[0141] The background was brightly lit (100 lux), while the stimulus was dark, resulting in different contrast levels. Therefore, the test was conducted as a photopic vision test rather than a twilight vision test. In some embodiments, the background may be dark and the stimulus as a brightly lit point. Each stimulus was presented for approximately 250 milliseconds, followed by a response waiting time of approximately 300 milliseconds. These periods were adjustable by a control program according to the subject's response speed. For example, they could be dynamically adjusted before or during the test based on pre-test confirmation. Generally, a stimulus size of 0.44 degrees was used with a central radius of 24 degrees. This corresponds to the standard Goldmann stimulus size III. The stimulus size in the peripheral region (radii from 24 to 40 degrees) was doubled to 0.88 degrees. Doubling the stimulus size in peripheral vision was intended to compensate for the degradation of the display lens performance in the peripheral region. Since normal human vision also deteriorates in the peripheral region, the impact of such lens degradation was significant. The test program was also designed to allow for changes in stimulus size to suit various patient cases.

[0142] The fixation targets (patterns) in Figure 35G were placed in the center of the screen for each eye being tested. These targets were constructed as multi-colored dots, rather than the monochromatic fixation points typically used in conventional Humphrey tests. This color variation made it easier to attract the subject's attention and focus on the target. The frequency of color changes was not synchronized with the frequency of stimulus appearance, preventing the subject from associating the two and responding incorrectly. This test protocol also allows for adjustment of the fixation target size to suit the patient's symptoms. Furthermore, an eye / pupil tracking system may be used to monitor the subject's eye fixation at various time intervals. The eye tracking system transmits the gaze vector direction to the test program, informing it whether the subject is properly focusing on the center.

[0143] Fixation was checked individually for each eye using pupil / gaze data. Pupil / gaze data was acquired at various timings, and if the gaze direction vector was approximately 0 degrees, the subject was focused on the central target; otherwise, the program paused and waited until the subject fixed again. If the patient was not fixing, the stimulus was not displayed, and the test stopped until the participant fixed again. Due to fine eye movements on the fixation target, an acceptable margin of error was allowed. Fixation was checked at two main timings for each stimulus location: before displaying each stimulus in the stimulus sequence (e.g., before each of the four stimulus contrast levels mentioned above), and before recording the response, regardless of whether the response was positive (e.g., the patient was looking at the stimulus) or negative (e.g., the patient was not looking at the stimulus). Negative responses were recorded at the end of the stimulus sequence period, in addition to the allowed response time. Checking for fixation before showing the stimulus sequence was to confirm that the patient was focused on the fixation target. If the subject was not fixating, the stimulus was not displayed, and the test was stopped until the participant fixed again.

[0144] Figure 36 is a timing diagram showing the processing of an examination sequence at a single stimulus location. In one example, the pupil tracking device may be separate from or part of the visual system or the device of the visual system, but may include an inward-facing image sensor and be configured to provide data to an image display device. The image display device may include a projector to change the position of the projected stimulus according to the movement of the gaze. In this way, even if the subject is looking around and not fixating, the stimulus may move with the subject's eyes, and the examination of the desired position in the visual field may continue. For this reason, rather than stopping the stimulus sequence when it is determined that the subject is focusing on something other than the fixation target, the stimulus sequence may be configured in the sequence to correspond to the intended position in the subject's visual field, so that the stimulus continues to change and is repositioned based on determining the subject's current fixation point.

[0145] For each subject, the visual field test began with an explanation of how the test would proceed. Eyeglasses were fitted to the patient to ensure they could clearly see the fixation target, and the target size was adjusted accordingly if necessary. Eye tracking calibration was performed at a single fixation target. Following this, a demonstration mode was presented to the subject. This mode followed the same sequence as the main test, but in this example, the number of positions was reduced to seven, and no responses were recorded. The purpose of this mode was to provide the subject with training regarding the test. This training mode also allowed the program operator to verify the accuracy of the eye tracking system, the patient's response speed, and the position of the patient's eyes relative to the headset, thus preventing errors or deviations in the main test.

[0146] Then, the normal blind spot was scanned by presenting suprathreshold stimuli placed at four different positions within a 15-degree proximity range, spaced 1 degree apart. This was a useful step in avoiding rotational mismatch between the headset and the subject's eyes.

[0147] Next, 52 stimulus sequences were presented to the patient in no particular order at pre-specified locations. Participants responded to the stimuli by activating an electronic clicker or making gestures. After recording the participant's responses at all locations, the locations of "missing" points were temporarily stored. A retrieval algorithm was then used to find the locations of all "seen" points on the boundary of the "missing" points. These two sets of points were then examined again to eliminate random response errors in the participants, ensuring that the visual field areas were continuous. False positive responses, false negative responses, and fixation loss (if present) were calculated and reported by the end of the test. As a result, all 52 responses were interpolated using the cubic method to generate a continuous visual field plot for the tested participants.

[0148] Visual field testing was conducted on 20 volunteer subjects using a simulated visual field defect created by covering a portion of the inner display lens of an eyeglass device. Results were evaluated point by point, using an image showing the covered area of ​​the display. As a criterion for testing accuracy, 52 responses were compared at approximately corresponding positions on the display image of the covered headset. The calculated errors are summarized in Table 1. [Table 1]

[0149] Meanwhile, visual field tests of 23 clinical patients were compared with the most recent results of Humphrey Field Analyzer (HFA) tests typically performed by the subjects during their consultations. The two types of visual field analyzers were compared by matching a common 24-degree central region. In this test as well, comparison and relative error calculation were performed point by point within the common 24-degree central region. Areas outside this region were judged based on their continuity with the central region and the absence of isolated response points. The calculated errors are summarized in Table 2. [Table 2]

[0150] Next, an image remapping process was performed. This involved identifying new dimensions and a center for the display image shown to the patient. The output image was then fitted into the subject's bright-field vision by resizing and shifting the original input image.

[0151] The visual field was binarized by setting all patient responses indicating "seen" to 1 and leaving "not seen" responses at 0. This resulted in a small 8x8 binary image. In other embodiments, the size of the binary image used may be increased or decreased. Small regions containing up to four connected pixels were removed from the binary visual field image. Four connected pixels represented a predetermined threshold for determining the small region, although in some embodiments, the threshold may be increased or decreased. Such small regions were not considered in the image embedding process. The ignored small regions represent either normal blind spots, insignificant defects, or any random error responses that may occur during the subject's visual field examination.

[0152] Based on the interpolated binary field image, the characteristics of the brightfield region were calculated. The characteristics calculated for the bright region included the following: 1) Bright region as a unit of pixels 2) Boxes that form the boundaries of the region 3) Centroid of the weighted region 4) The box that forms the list boundary of all pixels constituting the bright region within the field of view was defined as the smallest rectangle enclosing all pixels constituting the bright region. The centroid of the region was defined as the mass center of the region calculated with respect to the horizontal and vertical coordinates. The value of this characteristic corresponds to the new center of the output image and corresponds to the amount of image shift required for mapping.

[0153] By using the list of pixels that make up the largest bright field, as shown in FIG. 37, the width and height of all the pixels that define the boundary of the bright field were calculated. For each row of the bright field, two boundary pixels were identified, and their vertical coordinates were subtracted to obtain the width BF of the bright field in that row. , , , This width calculation was repeated for all rows that make up the bright field to calculate BF. widths A similar repetition may be applied to the per-column process for calculating BF. heights After that, either one of the two scaling equations may be used to newly determine the size, Width map and Height map of the mapped output image as shown in FIG. 37.

[0154] Width map may be calculated using the following size change equation.

Equation

[0155] BF [[ID=​​​​​​​​​​​​​​​

[0157] I size This is the size of the interpolated image (output image size), and BX widths BX heights θ is the width and height of the boundary box. The sum in the numerator of this equation is approximately equal to the brightfield area calculated for the horizontal and vertical directions, respectively. Therefore, dividing these sums by the square of the output image size gives an estimate of the proportional number of image areas to be mapped in each direction. These proportional numbers are multiplied by the corresponding dimensions of the boundary box that have already been calculated. The mapping behavior of this method is to fit the image into the maximum brightfield while maintaining the aspect ratio of the output image. This effect is achieved by incorporating the dimensions of the boundary box into the calculation. However, it cannot be said that the aspect ratio was maintained in all field patterns where defects were observed.

[0158] In one embodiment, the AI ​​system may use these two equations, as well as dozens, if not hundreds, of other equations in an optimization process, to determine which of the visible field of view can best fit into the image. Based on operator feedback, the system may learn to prefer one equation over the others based on the specific field of view to be corrected.

[0159] These remapping techniques were used in hazardous object identification tests. The remapping method was used on 23 subjects, using test images of safety-prone objects, including vehicles in this test. The test images were selected to examine the four main quadrants of the field of view, as shown in Figure 38. Using an example of the field of view, the test images displayed to the subjects were remapped. Subjects were tested by being shown images of approaching vehicles. Subjects could not see the vehicles until they were shown the remapped images. Figure 39A shows the image the subjects saw without remapping, and Figure 39B shows the image they saw after remapping. According to the inventors' prior research, 78% of subjects (18 out of 23) were able to identify safety problems that they could not identify without this assistive technology. Some subjects were tested individually for each eye, so 33 eyes were tested. In 23 of the 33 eyes, the visual assistive technology was found to be effective in identifying simulated approaching hazardous objects (P=0.023).

[0160] As noted, in some embodiments, with respect to Figure 1A, the examination subsystem 122 may determine one or more defective portions of the user's visual field based on the user's eye response to a set of stimuli presented to the user, or the lack of a response to a set of stimuli (a response being, for example, an eye movement response, a pupil size response, etc.). In some embodiments, one or more moving stimuli may be dynamically presented to the user as part of the visual examination presentation, and the response to or lack thereof to the stimuli may be recorded and used to determine which portions of the user's visual field are healthy. For example, in the dynamic portion of the visual examination presentation, recording of the patient's eye response may begin after the stimuli are presented in the visual examination presentation and continue until the stimuli disappear (for example, the stimuli may disappear after moving from the starting point to the center point of the visual examination presentation). In another example, during the visual examination presentation, when it is determined that the patient has recognized the stimuli (for example, when the patient's gaze direction changes to the current position of the stimuli), the stimuli may be removed (for example, disappear from the patient's field of view). In this way, the time required for presenting the visual test can be shortened and made more interactive (for example, giving the patient the feeling of playing a game rather than being diagnosed with a visual defect). Based on the aforementioned instructions (indicating a response to or lack thereof to a set of stimuli), the test subsystem 122 may automatically determine the defective portion of the user's visual field.

[0161] In some embodiments, the inspection subsystem 122 may determine one or more defective portions of the user's field of view, and the visual subsystem 124 may provide an enhanced image or adjust one or more configurations of the wearable device based on the determination of the defective portions. For example, the enhanced image may be generated or displayed to the user such that one or more predetermined portions thereof (e.g., regions of the enhanced image corresponding to the macular region of the user's eye's field of view or regions within the macular region of the eye) are outside the defective portions. As another example, the position, shape, or size of one or more display portions of the wearable device, the brightness, contrast, saturation, or sharpness level of the display portions, the transparency of the display portions, or other configurations of the wearable device may be adjusted based on the determined defective portions.

[0162] Figure 4 shows a process 400 illustrating implementation examples for both the examination mode and the subsequent visual mode. In block 402, the examination mode acquires data from a diagnostic device such as an image sensor embedded inside a mobile phone or tablet PC, or other user input device such as an eyeglass device. In block 404, the examination mode diagnosis may be performed to detect and measure ocular abnormalities from the received data (e.g., visual field defects, eye displacement, pupillary movement and size, or images of patterns reflected on the surface of the cornea or retina). In one example, the control program and algorithm were implemented using MATLAB R2017b (MathWorks, Inc., Natick, Massachusetts, USA). In various embodiments, the subject or examiner may be given the option of examining each eye individually or both eyes sequentially in one pass. In some embodiments, the examination mode may include an applied fast thresholding scheme that utilizes a stimulus sequence at a given location to include contrast step-like stimuli covering a central radius of 20 degrees or more. As an example, the examination mode may include an applied high-speed thresholding scheme that covers a central 40-degree radius using a sequence of 52 stimuli at a given location and includes four contrasting step-like stimuli. This is described herein with reference to Figures 35A to 35G and Figure 36. As another example, the examination mode may include automatically determining a visual defect (e.g., a defective field of vision) based on one or more responses of the user's eyes to a set of stimuli presented to the user, or the absence of such a response of the user's eyes to a set of stimuli (e.g., eye movement response, pupil size response, etc.), as described herein.

[0163] In block 406, the determined diagnostic data may be compared to a database or dataset containing modified profiles to compensate for identifiable ocular diseases (see, for example, Figure 16 and related explanations).

[0164] The identified correction profile may then be individually adjusted to compensate for, for example, differences in the visual axis, visual field defects, photosensitivity, diplopia, variations in image size between the two eyes, image distortion, and decreased visual acuity.

[0165] The personalized profile may be used by block 408 along with real-time data to process the image (e.g., using an image processor, scene processing module, and / or vision module). The real-time data may include data detected by one or more inward-facing image sensors 410 that provide pupil tracking data, and / or data from one or more outward-facing image sensors, including one or more field-of-view cameras 412 positioned to image a field-of-view screen. In block 414, real-time image correction may be performed, and the image may be displayed on the spectacle device as a regenerated digital image, or as an augmented reality image passing through the spectacle device so that the corrected portion is superimposed, or as an image projected onto the subject's retina (block 416). In some examples, the processing in block 414 is performed in combination with calibration mode 418. In calibration mode 418, the user may adjust the image correction using a user interface, such as an input device that allows the user to control the image and correction profile. For example, the user may shift the image of one eye laterally, upward, and downward, or rotate it (cyclotorted), in order to alleviate diplopia. In the above or other examples, the user may fine-tune the degree of field transformation (e.g., fisheye transformation, polynomial transformation, or conformal transformation) or translation, fine-tune the brightness and contrast, or invert the colors, in order to enlarge the field of view without adversely affecting visual function or causing unacceptable distortion.

[0166] Figure 5 shows an example of another process 500. Process 500 implements inspection mode and visual mode, similar to the example of process 400. In block 502, higher-order and lower-order aberration data are collected according to pupil size, degree of accommodation, and gaze. In some embodiments, all or part of this data may be collected from an aberration meter, or by capturing images of patterns or grid patterns projected onto the cornea and / or retina and comparing them to a reference image to detect aberrations of the corneal or entire ocular optics. The collected data may be sent to a visual correction framework. In block 504, the visual correction framework may determine an individualized correction profile, similar to block 406 described above. Blocks 508 to 518 perform functions similar to the corresponding blocks 408 to 418 of process 400.

[0167] Figure 8 shows a workflow 800 illustrating an examination module 802 that generates and presents multiple visual stimuli 804 to a user 806 using a spectacle device. User 804 has a user device 808 used for interaction to provide input responses to the examination stimuli. In some examples, the user device 808 may include a joystick, electronic scorer, keyboard, mouse, gesture detector / motion sensor, computer, smartphone or other telephone, dedicated device and / or tablet PC used by the user for interaction to provide input responses to the examination stimuli. The user device 808 may further include a processor and memory storing instructions. When executed by the processor, the instructions generate a GUI display for user interaction. The user device 808 may include memory, a transceiver (XVR) for sending and receiving signals, and an input / output interface for wired or wireless connection to a visual correction framework 810. The visual correction framework 810 may be stored in an image processing device. The visual correction framework 810 may be stored in the eyeglasses device, user device, etc., but in the illustrated example, the framework 810 is stored in an external image processing device. The framework 810 receives inspection mode information from the inspection module 802 and user input data from the user device 808.

[0168] Figure 9 shows the examination mode process 900 performed by workflow 800. In block 902, multiple examination stimuli are given to the subject according to the examination mode protocol. These stimuli may include patterns such as images of strings, images of objects, flashing lights, and grid patterns. The stimuli may be displayed to the subject or projected onto the subject's retina and / or cornea. In block 904, the visual correction framework may receive detection data from one or more inward-facing image sensors, such as data corresponding to the physical state of the pupil (e.g., visual axis, pupil size, and / or corneal margin). Block 904 may further include receiving user response data collected from the user in response to the stimuli. In block 906, the pupillary position state may be determined for multiple different stimuli. For example, this may be determined by measuring the difference in position and the difference in positional displacement between one stimulus and another different stimulus.

[0169] In block 908, astigmatism may be assessed across the entire visual field. This may include pupillary displacement data and / or analysis of ocular aberrations (e.g., projecting a reference image onto the retina and cornea and comparing the reflected image from the retinal or corneal surface to the reference image). In block 910, total ocular aberrations may be determined (e.g., by projecting a reference image onto the retina and / or cornea and then comparing the reflected image from the retinal or corneal surface to the reference image, as described in Figures 31A, 32 to 34 and related descriptions). In block 912, visual distortions such as optical distortions like coma, astigmatism, or spherical aberration, or visual distortions caused by retinal diseases, may be measured across the entire visual field. In block 914, visual field sensitivity may be measured across the entire visual field. In various embodiments of the process in Figure 9, one or more of blocks 904 to 914 may be performed optionally.

[0170] In some examples, the visual systems described herein can evaluate data obtained in examination mode to determine the type of ocular abnormality and the type of correction required. For example, Figure 10 illustrates process 1000, which includes an artificial intelligence correction algorithm mode that may be implemented as part of the examination mode. The machine learning framework is loaded in block 1002. Examples of frameworks may include dimensionality reduction, set learning, meta-learning, reinforcement learning, supervised learning, Bayesian methods, decision tree algorithms, linear classifiers, unsupervised learning, artificial neural networks, association rule learning, hierarchical cluster analysis, cluster analysis, deep learning, semi-supervised learning, etc.

[0171] In block 1004, the type of visual field defect is determined. Three examples of visual field defects are illustrated: an uncompensated blind spot field 1006, a low-sensitivity partial blind spot 1008, and a normal visual field 1010. After determining the visual field defect, block 1004 applies an appropriate correction protocol for the visual mode. For example, for the uncompensated blind spot field 1006, in block 1012, the visual correction framework tracks vision using pupil tracking with an inward-facing image sensor, etc., and performs video tracking of moving objects in the visual field (e.g., using an outward-facing image sensor such as an external camera). In the illustrated example, in block 1014, a safety issue located in or moving into the blind spot area is detected, for example, by comparing the location of the safety issue with the defective mapped visual field measured in inspection mode. In block 1016, the object of interest may be monitored at various positions, including central and peripheral positions.

[0172] In the example of a partial blind spot 1008, the enhanced visual acuity mode may be initiated in block 1018. From this block, objects in the visual field are monitored by tracking the central portion of the visual field. In block 1020, an image segmentation algorithm may be employed to separate the object from the visual field. Further enhanced contours may be applied to the object and displayed to the user. The contours coincide with the identified outer edges of the segmented object. For a normal visual field 1010, in block 1022, a customized correction algorithm may be applied to correct aberrations, visual field defects, esotropia, and / or visual distortion.

[0173] In some embodiments, the examination subsystem 122 may determine a plurality of correction profiles associated with the user (e.g., during a visual examination presentation, while an enhanced presentation of live image data is displayed to the user, etc.). In some embodiments, each correction profile may include a set of correction parameters or functions applied to live image data in a given context. As an example, the user may have correction profiles for each set of eye characteristics (e.g., range of gaze direction, pupil size, margin position, or other characteristics). As a further example, the user may additionally or alternatively have correction profiles for each set of environmental characteristics (e.g., range of ambient brightness levels, ambient temperature, or other characteristics).

[0174] Based on currently detected eye-related or environment-related characteristics, a corresponding set of modification parameters or functions may be obtained and used to generate an enhanced presentation of live image data. For example, the corresponding set of modification parameters or functions may be obtained (e.g., applied to an image to modify the image for the user) based on the fact that the currently detected eye-related characteristics match a set of eye-related characteristics associated with the obtained set of modification parameters or functions (e.g., the currently detected eye-related characteristics fall within the associated set of eye-related characteristics). In some embodiments, the set of modification parameters or functions may be generated based on currently detected eye or environment characteristics (e.g., ad-hoc generation of modification parameters, adjustment of a set of modification parameters or functions of a currently stored modification profile associated with a user for a given context).

[0175] In one use case, a wearable device (implementing the aforementioned behavior) may automatically adjust the brightness of an enhanced presentation of live image data to one or more of the user's eyes based on the respective pupil sizes (for example, such adjustments are independent of the ambient brightness). For example, subjects with unequal pupil size have uneven pupil sizes and possess monocular photosensitivity that prevents them from tolerating light levels that healthy eyes can tolerate. In this way, the wearable device can automatically adjust the brightness of each eye individually (for example, based on the detected pupil size of each eye).

[0176] In another use case, a wearable device may detect pupil size, visual axis, optical axis, marginal position, line of sight, or other accommodative states (e.g., including changes in the aforementioned states) and modify a correction profile based on the detected states. For example, in a subject with higher-order aberrations (e.g., refractive errors that cannot be corrected by glasses or contact lenses), the subject's aberrations are dynamic and change depending on pupil size and the ocular accommodative state. The wearable device may detect the accommodative state by detecting the appearance of the pupillary near reflex (e.g., miosis (the pupil becoming smaller) and accommodative convergence (the eye moving inward)). Furthermore, or alternatively, the wearable device may include pupil and line-of-sight tracking devices to detect the direction of the line of sight. In another example, ocular aberrations change depending on the size and position of the opening of the visual system and can be measured in relation to different pupil sizes and the position of the pupil and visual axis. The wearable device may, for example, determine higher-order aberrations by measuring the surface irregularities on the cornea (e.g., based on the measurements) and calculate a correction profile to address the higher-order aberrations. Different correction profiles may be created for different pupil and visual axis sizes and positions (or other accommodative states of the eye), saved for future use, and used to provide real-time augmentation. One or more of these detected inputs allow the wearable device to perform augmentation for the user using the appropriate correction profile (e.g., a set of correction parameters or functions).

[0177] As another example, a wearable device may be used to correct presbyopia. In this case, the wearable device provides near vision by automatically performing autofocus on the image displayed to the user. To further enhance and improve near vision, the wearable device may detect where the user is trying to look at a near object (for example, by detecting the behavior of near reflexes such as miosis (smallening of the pupil) and accommodative convergence (movement of the eyes inward)) and perform autofocus on the area of ​​the image corresponding to the object the user is looking at (for example, the part of the display the user is looking at, or the near area around the object the user is looking at). Furthermore, or alternatively, the wearable device may determine how far away a target (for example, a target object or area) is by quantifying the amount of near reflexes exhibited by the subject and the distance from the eyes to the target (for example, via sensors in the wearable device), and provide appropriate corrections based on the quantified amount and the target distance.

[0178] As another example, a wearable device may be used to correct diplopia (e.g., associated with strabismus). The wearable device may monitor the user's eyes, track the user's pupils to measure their deviation, and displace the image projected onto each eye (in conjunction with detecting symptoms such as strabismus). Because diplopia is generally dynamic (e.g., it increases or decreases towards one or more lines of sight), the wearable device can provide appropriate correction by monitoring the user's pupils and lines of sight. For example, if a user has trouble moving their right pupil away from their nose (e.g., towards the edge of their face), their diplopia may increase when they are looking to the right and decrease when they are looking to the left. In this way, the wearable device displays an enhanced presentation of live image data to each eye, and the first version of the enhanced presentation displayed to one eye of the user reflects the displacement from the second version of the enhanced presentation displayed to the other eye of the user (for example, the amount of displacement is based on pupil position and gaze direction), thereby dynamically compensating for the user's condition (e.g., strabismus or other conditions), and consequently preventing diplopia in any gaze direction.

[0179] Prisms can be applied to correct diplopia by displacing an image presented in front of an eye with strabismus (e.g., caused by strabismus or other conditions), but prisms cannot cause image torsion and are therefore not useful for correcting diplopia resulting from conditions that cause images to appear tilted or rotated (cyclotorted) (e.g., rotational strabismus is a form of strabismus in which images received by both eyes appear tilted or rotated). In some use cases, a wearable device can measure the degree of strabismus (e.g., including rotational conditions (cyclotorsion)) by monitoring the user's eyes and correlatively detecting the pupils, margins, lines of sight, or visual axes of both eyes. Alternatively, such measurements by a wearable device may be performed by acquiring images of the retinas of both eyes and comparing the retinal and nerve structures with each other. In this case, the wearable device may detect and measure the relative positions of the structures of those eyes and any torsional displacements. Such measurements may be provided to predictive models for predicting correction parameters for live image processing to correct defects and mitigate diplopia. Continuous feedback from sensors in the wearable device (e.g., pupil tracking devices, eye-tracking devices, retinal image-based tracking devices, etc.) may be used to modify the correction profile applied to the live image data in real time. In further use cases, the user may also fine-tune the correction. For example, an image may be displayed to the user on a user interface, and the user may move the image (or an object represented by the image) (e.g., using an input device such as a joystick) until the image crosses in front of one eye, and rotate the object until it overlaps with the image seen by the other eye. In some embodiments, when signs of diplopia are detected, the wearable device may perform diplopia-related measurements or corrections by automatically moving (e.g., translating or rotating) the image as it crosses in front of one eye, without explicit user input indicating that the image should be moved or the amount and location of the movement.

[0180] As with other forms of strabismus, the resulting displacement due to rotational strabismus changes in real time based on the intended direction of action of the paralyzed (or partially paralyzed) muscles associated with rotational strabismus, and changes when such a patient is looking to one side or the other. By tracking the characteristics of the eyes, a wearable device can dynamically compensate for the user's symptoms. In this way, the wearable device displays an enhanced presentation of live image data to each eye, so that the first version of the enhanced presentation displayed to one eye of the user reflects the displacement from the second version of the enhanced presentation displayed to the other eye of the user (for example, so that the amount of displacement is based on pupil position and gaze direction).

[0181] In some embodiments, with respect to Figure 1A, if the test subsystem 122 obtains feedback related to a set of stimuli (displayed to the user during a visual test presentation), feedback related to one or more of the user's eyes, feedback related to the user's environment, or other feedback, it may provide this feedback to a predictive model, which may be configured based on the feedback. In some embodiments, the test subsystem 122 may obtain a second set of stimuli (e.g., during a visual test presentation). As an example, the second set of stimuli may be generated based on the processing of the set of stimuli by the predictive model and the feedback related to the set of stimuli. The second set of stimuli may be additional stimuli obtained from the feedback to further examine one or more other aspects of the user's visual field (e.g., to facilitate more granular correction or other enhancement to the user's visual field). In one use case, the test subsystem 122 may display the second set of stimuli to the user (e.g., during the same visual presentation) and, in response, obtain further feedback related to the second set of stimuli (e.g., further feedback indicating whether or how the user is looking at one or more stimuli in the second set). The examination subsystem 122 may then provide the predictive model with further feedback relating to a second set of stimuli, and the predictive model may be further configured based on the further feedback (e.g., during the presentation of the visual test). As an example, the predictive model may be automatically configured for the user based on (i) a display of the user's response to one or more stimuli (e.g., from the set of stimuli, from the second set of stimuli, or from any other set of stimuli), (ii) a display of the user's lack of response to such stimuli, (iii) an eye image captured during the presentation of the visual test, or other feedback (e.g., the predictive model may be personalized for the user based on feedback from the presentation of the visual test). In one use case, for example, the feedback may indicate one or more visual defects in the user, and the predictive model may be automatically configured based on the feedback to address the visual defects.As another example, a predictive model may be trained on such feedback and other feedback from other users to improve the accuracy of the results it provides (for example, being trained to provide the correction profiles described herein or to generate an enhanced presentation of live image data).

[0182] In some embodiments, the visual subsystem 124 may provide live image data or other data (e.g., monitored eye-related characteristics) to a predictive model to acquire and display an enhanced image (obtained from the live image data). In some embodiments, the predictive model may remain configured during the display of the enhanced image (obtained from the live image data) based on further feedback continuously provided to the predictive model (e.g., periodically, according to a schedule, or based on other automated triggers). As an example, a wearable device may acquire a live video stream from one or more cameras of the wearable device and display an enhanced image on one or more displays of the wearable device (e.g., within a range of less than 1 / 1000th of a second, less than 1 / 100th of a second, less than 1 / 100th of a second, less than 1 second, etc., from the time the live video stream is captured by the wearable device's cameras). In some embodiments, the wearable device may acquire an enhanced image from the predictive model (e.g., in response to providing the predictive model with live image data, monitored eye-related characteristics, or other data). In some embodiments, a wearable device may obtain modification parameters or functions from a predictive model (e.g., in response to providing the predictive model with live image data, monitored eye-related characteristics, or other data). The wearable device may use modification parameters or functions (e.g., parameters of a function used to convert or modify live image data into an enhanced image) to generate an enhanced image from live image data. As a further example, the modification parameters may include one or more conversion parameters, brightness parameters, contrast parameters, saturation parameters, sharpness parameters, or other parameters.

[0183] For example, an automated personalized correction profile may be created by using a visual correction framework with a machine learning framework that includes an AI algorithm to apply visual field transformation, translation, and resizing to better fit the visual field to the remaining functional visual field. The machine learning framework may include one or more of the following: data collection, visual field classification, and / or regression models. A graphical user interface (GUI) and data collection program may be used to facilitate recording of participant responses, quantitative scores, and feedback.

[0184] Regarding transformations applied to images in visual mode, exemplary transformations in the machine learning framework may include one or more of the following: 1) conformal mapping, 2) fisheye, 3) custom quartic polynomial transformation, 4) polar polynomial transformation (using polar coordinates), or 5) rectangular polynomial transformation (using rectangular coordinates) (e.g., Figure 13).

[0185] Regarding translational movements applied to an image in visual modes, one or more of the following examples may be included. For center detection, the best center and a weighted average of the nearest neighbors to the center may be used. For example, the nearest neighbor may be determined by finding the point closest to the center position. The best center may be determined by one or more of the following: 1) the centroid of the largest component, 2) the center of the largest inscribed circle, largest inscribed square, largest inscribed rhombus and / or largest inscribed rectangle, or 3) the center of the locally largest inscribed circle, inscribed square, inscribed rhombus and / or inscribed rectangle (e.g., Figure 14). For example, the framework searches for the largest shape, but instead, the framework may employ a weighted average of the nearest neighbors using multiple methods so as not to stray from the macular visual region.

[0186] In various embodiments, the AI ​​algorithm may first be trained using simulated visual field defects. For example, a dataset of visual field defects may be collected to train the AI ​​algorithm. For instance, in one experimental protocol, a dataset of 400 visual field defects was obtained from glaucoma patients. Using this dataset, simulations of visual field defects may be performed on virtual reality glasses to be presented to normal subjects for rating. The algorithm may then be trained using the feedback obtained from this rating.

[0187] For example, an AI algorithm may be used that automatically fits the input image into areas corresponding to a normal visual field pattern for each patient. In various embodiments, this algorithm may have at least three degrees of freedom in remapping the image, but the number of degrees of freedom used may be increased or decreased. In one example, the degrees of freedom include transformation, shifting, and resizing. The additional image transformation may maintain the quality of the central region of the image corresponding to the central visual acuity, which is the part with the highest visual acuity, while condensing the peripheral regions, which have moderate image quality. This may be applied so that the generated image is fully perceptible to the patient.

[0188] The image transformations included in the AI ​​algorithm may include one or more of conformal mapping, polynomial transformation, or fisheye transformation. In some embodiments, other transformations may be used. The machine learning technique may be trained on a marked dataset before actually performing the task. In one example, the AI ​​algorithm may be trained on a field of view dataset containing various types of peripheral defects. For example, in one experiment, the dataset used contained 400 field of view defect patterns. In the training phase, normal participants then assigned quantitative scores to the remapped images created by the AI ​​algorithm.

[0189] Figure 11 shows an example of a test image (stimulus) image 1100. The test image 1100 may be designed to measure visual acuity, paracentral visual acuity, and / or peripheral visual acuity. The illustrated test image, as shown in Figure 11, displays five letters in the central region, four inner diamonds 1102 in the paracentral region, and eight outer diamonds 1104 in the peripheral region.

[0190] As mentioned above, a large amount of data is necessary to train an AI system. As a first step, the patient's binocular vision may be simulated using the binocular field of view including the blind spot, as shown in Figure 12. Subsequently, the simulated visual acuity may be presented to the subject using eyeglasses. In this way, the input image can be manipulated using various image manipulations, and then presented again to the subject for rating the modified appearance. The modification process may be continued, and the modified image may be further modified and presented to the subject until the optimal modified image is determined. Figure 13 shows examples of several different modification transformations applied to an image to be presented to the user. Figure 14 shows examples of several different translation methods (shifting the image to fit it into the normal field of view). The normal field of view is white, and the blind spot is black.

[0191] The AI ​​system may be designed using machine learning models such as artificial neural networks and support vector machines (SVMs). In some examples, the AI ​​system is designed to generate an output that includes the estimation of the best image manipulation method (e.g., geometric transformation and translation) by an optimization AI system. In the visual mode, the visual system presents the patient with an image manipulated according to the output image manipulation method using a headset, and may present the image in a way that gives the patient the best possible vision, based on its own field of vision, including any defects. The machine learning framework of the visual correction framework (also referred to herein as the "AI system") may be trained using collected data (e.g., as described herein). A block diagram showing an example of AI system 1500 is shown in Figure 15.

[0192] Process 1600 of AI system 1500 is shown in Figure 16. Input to system 150 includes examination images and visual field images. AI system 1500 estimates which geometric transformations are best to perform on the examination images to present more detailed content across the visual field. AI system 1500 then estimates the best translational movement for the examination images so that the displayed image covers most of the visual field. After this, the examination images are transformed and translated, as shown in Figures 17 and 18, respectively. Finally, in the case of training for simulation purposes only, this image is combined with the visual field again, but in the examination phase, it is displayed directly to the patient. Figure 19 shows a screenshot of a graphical user interface presenting a summary of the visual field analysis, including a final implementation example of the visual field AI system with parameters for image transformation and translational movement to be applied to the images.

[0193] In the implementation example, a machine learning framework ("AI system") is implemented in the visual correction framework using an artificial neural network model. The AI ​​system acquires a field of view image converted to a vector. As output, the AI ​​system provides predicted results of image transformation and translation parameters to be applied to the scene image. The scene image is then manipulated using these parameters. The AI ​​system has two hidden layers. Each hidden layer has three neurons (i.e., units) and one output layer. An example of such an AI system model is shown in Figure 20. In other examples, this AI system may be extended to a convolutional neural network model to obtain even more accurate results. Figures 21 and 22 show, as examples, process 2100, which is the result of applying a neural network to inspection mode, and process 2200, which is an AI algorithm optimization process using a neural network, respectively.

[0194] In some embodiments, with respect to Figure 1A, when the examination subsystem 122 obtains feedback relating to a set of stimuli (displayed to the user during the visual examination presentation), feedback relating to one or more of the user's eyes, feedback relating to the user's environment, or other feedback, it may provide this feedback to a predictive model, which may be configured based on the feedback. In some embodiments, further feedback may be continuously obtained and provided to the predictive model (for example, periodically, according to a schedule, or based on other automated triggers) to update the configuration of the predictive model. As an example, the configuration of the predictive model may be updated while one or more images enhanced with live image data are displayed to the user.

[0195] In some embodiments, the visual subsystem 124 may monitor characteristics related to one or more of the user's eyes (e.g., gaze direction, pupil size or response, margin position, visual axis, optical axis, eyelid position or movement, head movement, or other characteristics) and provide eye characteristic information to the predictive model during the enhanced presentation of live image data to the user. Furthermore, or alternatively, the visual subsystem 124 may monitor characteristics related to the user's environment (e.g., ambient brightness level, ambient temperature, or other characteristics). As an example, based on eye or environmental characteristic information (e.g., indicating monitored characteristics), the predictive model may provide one or more modification parameters or functions to apply to the live image data to generate an enhanced presentation of the live image data (e.g., presentation to the user of one or more enhanced images derived from the live image data). In one use case, the predictive model may retrieve modification parameters or functions (e.g., stored in memory or one or more databases) based on currently detected eye or environmental characteristics. In another use case, the predictive model may generate modification parameters or functions based on currently detected eye or environmental characteristics.

[0196] In some embodiments, with respect to Figure 1A, the visualization subsystem 124 may facilitate the enhancement of the user's field of vision via one or more dynamic display portions on one or more transparent displays (for example, based on feedback related to a set of stimuli displayed to the user or other feedback). For example, the dynamic display portion may include one or more transparent display portions and one or more other display portions (for example, display portions of a wearable device or other device). In some embodiments, the visual subsystem 124 may cause one or more images to be displayed on the other display portions (for example, so that the images are not displayed on the transparent display portions). For example, the user may be able to see through the transparent display portions of the transparent display but not the other display portions, and instead see image presentations on the other display portions of the transparent display (for example, portions around or adjacent to the transparent display portions). That is, in some embodiments, a dynamic hybrid see-through / opaque display can be used. In this way, for example, one or more embodiments can (i) avoid the bulky size and heavy weight of typical virtual reality headsets, (ii) utilize the user's healthy vision (e.g., if central vision is healthy but peripheral vision is impaired, utilize the good central vision; if peripheral vision is healthy but central vision is impaired, utilize the healthy peripheral vision), and (iii) mitigate the visual confusion that would be caused by typical augmented reality technologies where there is an overlapping effect between the see-through scene and the scene displayed inside.

[0197] For example, live image data may be acquired via a wearable device, and enhanced images may be generated based on the live image data and displayed on other display areas of the wearable device (e.g., display areas of the wearable device's display that either meet the opacity threshold or the transparency threshold). In some embodiments, the visual subsystem 124 may monitor one or more changes related to one or more of the user's eyes and adjust the transparent display area of ​​the transparent display based on the monitoring. For example, the monitored changes may include eye movements, changes in gaze direction, changes in pupil size, or other changes. The position, shape, size, transparency, brightness level, contrast level, sharpness level, saturation level, or other aspects of one or more transparent display areas or other display areas of the wearable device may be automatically adjusted based on the monitored changes.

[0198] In one use case, with respect to Figure 24A, the wearable device 2400 may include a transparent display 2402 dynamically configured to have a transparent peripheral portion 2404 and an opaque central portion 2406, so that light from the user's environment can pass through the transparent peripheral portion 2404 but not through the opaque central portion 2406. In the case of a patient with a diagnosed central visual field defect 2306, the aforementioned dynamic configuration allows such a patient to see the actual, uncorrected scene of the environment using their healthy peripheral vision, while also being presented with a corrected depiction of the central region on the opaque central portion 2406.

[0199] In another use case, with respect to Figure 24B, the wearable device 2400 may include a transparent display 2402 dynamically configured to have an opaque peripheral portion 2414 and a transparent central portion 2416, so that light from the user's environment can pass through the transparent central portion 2416 but not through the opaque peripheral portion 2414. In the case of a patient with peripheral vision impairment, the aforementioned dynamic configuration allows such a patient to see the actual, uncorrected scene of the environment using their healthy central vision, while also being presented with a corrected depiction of the peripheral area on the opaque peripheral portion 2414. In each of the above use cases, with respect to Figures 24A and 24B, one or more positions, shapes, sizes, transparency, or other aspects of the transparent display portion 2404, 2416 or the opaque display portion 2406, 2414 may be automatically adjusted based on changes related to one or more of the user's eyes monitored by the wearable device 2400 (or other components of System 100). Furthermore, or alternatively, one or more brightness levels, contrast levels, sharpness levels, saturation levels, or other aspects of the opaque display portions 2406, 2414 may be automatically adjusted based on changes related to one or more of the user's eyes monitored by the wearable device 2400. In some cases, for example, to dynamically respond to areas of reduced brightness in the user's field of view, the user's pupils and gaze (or other eye characteristics) may be monitored and used to adjust the brightness levels of each portion of the opaque display portions 2406, 2414 (in addition to, or instead of, increasing the brightness level of each portion of the enhanced image corresponding to areas of reduced brightness in the user's field of view).

[0200] As an example, with respect to Figure 24C, based on the determination of the user's field of view (including, for example, defective and healthy fields of view, as represented by the field of view plane 2432), an enhanced image may be generated as described herein (for example, as represented by the remapped image plane 2434). The enhanced image may be displayed to the user on one or more opaque display portions within the opaque region 2438 of the display (for example, as represented by the selectively transparent screen plane 2416), thereby enhancing the user's field of view as they view the environment through the transparent region 2440 of the display.

[0201] In one use case, with respect to Figure 24C, the selectively transparent screen plane 2436 may be aligned with other planes 2432 and 2434 via one or more eye-tracking techniques. For example, the eye-tracking system (e.g., of a wearable device 2400 or other device) may be calibrated for the user to ensure appropriate image projection according to the individual user's healthy field of vision. The eye-tracking system may continuously acquire gaze coordinates (e.g., periodically, according to a schedule, or according to other automated triggers). It may also perform a coordinate transformation to convert the spherical coordinates (θ, φ) of eye movements to the Cartesian coordinates (x, y) of the display. Thus, the device controller may determine the center position of the displayed image. The camera image is cropped and shifted to match the acquired gaze vector direction (e.g., Figure 24C). The same Cartesian coordinates may be sent to the selectively transparent screen controller to make the area corresponding to macular visual acuity in the current gaze direction transparent, enabling the use of central vision. In some cases, low-pass filtering may be applied to gaze data to remove minute eye movements that cause image flickering in the user's view (for example, minute eye movements caused by constant movement during fixation or drafting, as the eyeballs never remain completely still).

[0202] As shown above, in some embodiments, the wearable device may be configured to selectively control the transparency of the display area of ​​a monitor, such as a screen, glass, film, and / or multilayer media. Figure 23 shows an exemplary process 2300 that implements an inspection mode and a visual mode, and the use of a custom reality glasses device, which may use macular (central) versus peripheral visual manipulation.

[0203] In some examples, a custom reality glasses device (e.g., Figures 40A to 40C) includes transparent glasses that overlay a corrected image onto the visible scene. The glasses may constitute a monitor including a screen, the screen of which transparency is controllable and for projecting the image to be displayed. For example, such a display includes a head-up display. In various embodiments, a custom reality glasses device comprises glasses having multiple controllable layers. These layers overlay a corrected image onto the scene seen through the glasses. The layers may be formed of glass, ceramic, polymer, film and / or other transparent materials and may have a multilayer structure. The controllable layers may include one or more electrically controllable layers whose transparency can be adjusted in one or more portions of the field of view by addressing pixels, for example. In one embodiment, this may include pixels or cells that are individually addressable (e.g., using electric current, electric field or light). The controllable layer may be a layer controlled to adjust the contrast of one or more portions of the field of view, the color filter of each portion, the zoom in / zoom out of each portion, the focus of each portion, and the transparency of the surface of the glasses device that displays the image in order to block or pass light incident from the external environment at specific locations in the field of view. If there is a portion of the field of view that is to be manipulated to enhance the subject's vision (for example, the peripheral vision portion or the macular vision portion or a portion that is partly macular and partly peripheral), the transparency of that portion of the glasses may be reduced to block the external environment from being seen through that portion of the glasses, and the manipulated image displayed in that portion of the glasses may be seen more clearly by the patient. In various embodiments, the visual system or custom reality glasses device may dynamically control region-specific transparency so that the subject has a natural vision of the external environment when the direction of the eyes changes not only due to head movement but also due to eye movement. For example, pupil tracking data (e.g., pupil and / or gaze tracking) may be used to modify the portion of the glasses that has decreased transparency so that the area with decreased transparency translates relative to the subject's eye.

[0204] For example, the transparency of the glasses included in the glasses device that constitutes custom reality glasses can be adjusted, but the adjustment may be controllable to block light from the portion of the field of view corresponding to the area where image modification is performed (e.g., the central or peripheral region). Without such adjustment, the subject would see the manipulated image and perceive the actual field of view in this region superimposed on the manipulated image. To block light in this way, the glasses device may be provided with a photochromic glass layer. Furthermore, the glasses device may change the location of the area where the transparency of the glasses is reduced by measuring eye (pupil) movement using an inward-facing image sensor and compensating based on such movement by processing with a visual correction framework. For example, a monitor display screen includes pixels or cells including electro-ink technology. These pixels or cells may be individually addressable, and an electric field may be generated to change the ink configuration within the cell, thereby changing the transparency and / or generating pixels on the display. In an implementation example, Figure 40A shows custom reality glasses 4000, which consist of a frame 4002 and two transparent glasses assemblies 4004. As shown in Figures 40B and 40C, the transparent glasses assembly 4004 has an electronically controllable correction layer 4006 embedded within it. The correction layer 4006 may be controllable from a completely transparent state to a completely opaque state and may be a digital layer capable of generating a correction image to be superimposed on or replaced on a portion of the field of view of the glasses 4000. The correction layer 4006 may be connected to an image processing device 4010 on the frame 4002 via an electrical connection 4008.

[0205] Referring specifically to process 2300 in Figure 23, block 2302 may involve receiving inspection mode data in a visual correction framework, and block 2304 may involve determining visual field distortions, defects, aberrations and / or other ocular abnormalities along with their locations.

[0206] Regarding the diagnosed central visual field anomaly 2306, in block 2308, the custom reality glasses device may allow images from the external environment to pass through the glasses and reach the user's peripheral visual field (e.g., Figure 24). As illustrated, the custom reality glasses device 2400 may include a multilayer glasses viewfinder 2402. The peripheral region 2404 may be set to be transparent so that light can pass through and the subject can see the actual, uncorrected external environment. In block 2312, the central region 2406 of the external environment may be made opaque by the glasses device 2400, and for the central region, a corrected depiction may be presented to the user by the display using corrections such as those shown in Figures 13, 14, 17, and 18.

[0207] Regarding the diagnosed peripheral vision abnormality 2308, in block 2314, the central region 2416 of the external environment (e.g., Figure 24B) can pass through the transparent portion of the spectacle device 2400. Then, the transparency of the peripheral region 2414 is changed to block light so that the corrected peripheral region image is displayed within the peripheral region 2414 by using, for example, the corrective transformation described herein.

[0208] In some embodiments, with respect to Figure 1A, the visual subsystem 124 can facilitate enhancement of the user's field of view via projection onto a selected portion of the user's eye (e.g., based on feedback related to a set of stimuli displayed to the user or other feedback). As described herein, an enhanced presentation of live image data may be displayed to the user by projecting the enhanced presentation (e.g., a modified image derived from live image data) onto the user's eye. In addition to, or instead of, using dynamic display portions on the screen (e.g., so that the user can look through one or more portions of the screen and see the modified live image data displayed on one or more other portions of the screen), the modified image data can be projected onto one or more portions of the user's eye (e.g., one or more portions of the user's retina) while simultaneously avoiding projection of the modified image data onto one or more other portions of the user's eye (e.g., one or more other portions of the user's retina).

[0209] In some embodiments, the modified image data can be projected onto one or more healthy portions of the user's field of vision while simultaneously avoiding projection of the modified image data onto one or more other healthy portions of the user's field of vision. For example, with respect to other healthy portions of the field of vision where projection of the modified image data is avoided, light from the user's environment can pass through the user's retina (e.g., without significant interference from light emitted by a projector), thereby allowing the user to see the environment through such other healthy portions of the field of vision. On the other hand, with respect to the healthy portions of the field of vision onto which the modified image data is projected, the projected light prevents the user from seeing the environment through the healthy projected portions of the user's field of vision. Nevertheless, by projecting the modified live image data onto healthy portions of the user's field of vision, the system makes it possible to use the modified live image data to enhance the user's field of vision (this is done, for example, in a similar manner to the use of dynamic display portions to enhance the user's field of vision).

[0210] In some embodiments, the visual subsystem 124 can monitor one or more changes related to one or more of the user's eyes and, based on the monitoring, perform adjustments to one or more projection parts of the projector (e.g., parts including laser diodes or LED diodes that emit light at a threshold visible to the user's eyes). For example, the monitored changes may include eye movements, changes in gaze direction, changes in pupil size, etc., as well as adjustments to dynamic display parts on a screen. One or more positions, shapes, sizes, brightness levels, contrast levels, sharpness levels, saturation levels, or other aspects of the projection parts or other parts of the projector may be automatically adjusted based on the monitored changes.

[0211] In one use case, the wearable device may include a projector configured to selectively project an enhanced presentation (e.g., a modified image derived from live image data) onto one or more parts of the user's eye (e.g., one or more parts of each of the user's retinas corresponding to the user's healthy field of vision) while simultaneously avoiding projecting the modified image data onto one or more other parts of the user's eye (e.g., one or more other parts of each of the user's retinas corresponding to the user's healthy field of vision). In some cases, the alignment of such selective projection planes may be aligned with other planes (e.g., a field of view plane, a remapped image plane, etc.) via one or more eye-tracking techniques (e.g., one or more techniques similar to those described in Figure 24C with respect to the use of dynamic display portions on a screen).

[0212] With respect to Figure 24A, the wearable device 2400 may include a transparent display 2402 dynamically configured to have a transparent peripheral portion 2404 and an opaque central portion 2406, so that light from the user's environment can pass through the transparent peripheral portion 2404 but not through the opaque central portion 2406. In the case of a patient with a diagnosed central visual field defect 2306, the aforementioned dynamic configuration allows such a patient to see the actual, uncorrected scene of the environment using their healthy peripheral vision, while also being presented with a corrected depiction of the central region on the opaque central portion 2406.

[0213] In another use case, with respect to Figure 24B, the wearable device 2400 may include a transparent display 2402 dynamically configured to have an opaque peripheral portion 2414 and a transparent central portion 2416, so that light from the user's environment can pass through the transparent central portion 2416 but not through the opaque peripheral portion 2414. In the case of a patient with peripheral vision impairment, the aforementioned dynamic configuration allows such a patient to see the actual, uncorrected scene of the environment using their healthy central vision, while also being presented with a corrected depiction of the peripheral area on the opaque peripheral portion 2414. In each of the above use cases, with respect to Figures 24A and 24B, one or more positions, shapes, sizes, transparency, or other aspects of the transparent display portion 2404, 2416 or the opaque display portion 2406, 2414 may be automatically adjusted based on changes related to one or more of the user's eyes monitored by the wearable device 2400 (or other components of System 100). Furthermore, or alternatively, one or more brightness levels, contrast levels, sharpness levels, saturation levels, or other aspects of the opaque display portions 2406, 2414 may be automatically adjusted based on changes related to one or more of the user's eyes monitored by the wearable device 2400. In some cases, for example, to dynamically respond to areas of reduced brightness in the user's field of view, the user's pupils and gaze (or other eye characteristics) may be monitored and used to adjust the brightness levels of each portion of the opaque display portions 2406, 2414 (in addition to, or instead of, increasing the brightness level of each portion of the enhanced image corresponding to areas of reduced brightness in the user's field of view).

[0214] In some embodiments, the testing subsystem 122 may monitor one or more eye-related characteristics related to the user's eye during a visual test presentation via two or more user interfaces (e.g., on two or more displays) and determine visual defect information for one or more of the user's eyes based on the eye-related characteristics that occur during the visual test presentation. For example, the testing subsystem 122 may cause one or more stimuli to be presented at one or more locations on at least one of the user interfaces and generate visual defect information for the user's eye based on one or more eye-related characteristics of the eye that occur at the time of stimulus presentation. In some embodiments, an eye deviation measurement may be determined based on an eye-related characteristic (as indicated by monitoring as occurring at the time of stimulus presentation) and used to provide correction or other enhancement for that eye. For example, the deviation measurement indicates the deviation of the eye relative to the other eye, and the deviation measurement may be used to determine and correct diplopia or other visual defects. For example, the amount of movement indicates the amount of strabismus (e.g., crossed eyes), and the direction (or axis) of movement indicates the type of crossed eyes. For example, if the eye movement was from "out" to "in," it means that the crossed eyes are outward-directed (e.g., exotropia). Thus, in some embodiments, double vision can be autonomously detected and corrected via a wearable device.

[0215] In some embodiments, the testing subsystem 122 may, when determining the deviation measurement of the user's first eye or other visual defect information, (i) present a stimulus at a location on the first user interface for the first eye, while ensuring that the stimulus intensity on the second user interface for the user's second eye does not meet a stimulus intensity threshold, and (ii) determine the visual defect information based on one or more eye-related characteristics of the first eye that occur at the time of stimulus presentation. For example, stimulus presentation on the first user interface may occur while no stimulus is presented on the second user interface. In one use case, if the first eye (e.g., the right eye) is exotropic immediately before such stimulus presentation on the first user interface (e.g., Figure 25D), by presenting the stimulus only in front of the first eye (e.g., only the right eye), the first eye will instinctively move toward and fixate on the stimulus location (e.g., within 1 second) because the second eye (e.g., the left eye) loses the advantage it had as a result of having nothing to look at. The testing subsystem 122 may measure corrective movements of the first eye (and other changes in the eye-related properties of the first eye) to determine the deviation measurement of the first eye. For example, the amount of movement of the first eye that occurs during such stimulus presentation may correspond to the amount of strabismus of the first eye.

[0216] In some embodiments, the testing subsystem 122 may, when determining the deviation measurement of the user's first eye or other visual defect information, (i) present a stimulus at a predetermined time at corresponding positions on the first user interface for the first eye and corresponding positions on the second user interface for the second eye, and (ii) determine the visual defect information based on one or more eye-related characteristics of the first eye that occur when the stimulus is presented. For example, the target stimulus may be presented at a central position on both user interfaces, or at another corresponding position on both user interfaces. In one use case, when a stimulus is presented in front of both eyes (e.g., Figure 25B), the dominant eye (e.g., the left eye in Figure 25B) instinctively moves to the corresponding position and fixates on the target stimulus (e.g., within 1 second). The other eye (e.g., the right eye in Figure 25B) also moves, but because this eye has exotropia, it does not instinctively fixate on the target stimulus, and the user experiences double vision. For example, the other eye moves instinctively, but this instinctive movement causes the gaze direction of the other eye to be directed to a different position. However, as the user focuses on viewing the target stimulus with the other eye, the other eye moves and fixates on the target stimulus presented at the corresponding position on the user interface for the other eye. Since the target stimulus is presented at the corresponding positions on both user interfaces, the dominant eye remains dominant and continues to fixate on the target stimulus presented at the corresponding position on the user interface for the dominant eye. The testing subsystem 122 may measure the corrective movement of the other eye (and other changes in the eye-related properties of the other eye) to determine the deviation measurement of the other eye (for example, the amount of movement of the other eye may correspond to the amount of strabismus of the other eye).

[0217] In some embodiments, after obtaining the user's first eye deviation measurement or other visual defect information by measuring changes in the eye-related characteristics of the first eye (e.g., the movement of the first eye when a stimulus is presented at a corresponding position on the first user interface for the first eye), the testing subsystem may cause the stimulus to be presented at a modified position on the first user interface for display to the first eye. For example, stimulus presentation at the modified position occurs while no stimulus is presented on the second user interface for the second eye (or while the stimulus intensity of the second user interface does not meet a stimulus intensity threshold, so that the second eye does not respond to any stimulus on the second user interface). Based on the fact that one or more eye-related characteristics of the first eye or the second eye do not change beyond a change threshold when presented at the modified position, the testing subsystem 122 may determine the first eye deviation measurement or other visual defect information. For example, the first eye deviation measurement may be determined based on the fact that the first eye does not move beyond a motion threshold (e.g., no motion or other motion threshold) when the stimulus is presented at the modified position. Furthermore, or alternatively, the deviation measurement of the first eye may be determined based on the fact that the second eye does not move beyond a motion threshold.

[0218] In some embodiments, the inspection subsystem 122 may generate one or more correction profiles associated with the user based on one or more deviation measurements of one or more of the user's eyes or other visual defect information (e.g., obtained through one or more visual inspection presentations). For example, each correction profile may include correction parameters or functions used to generate enhanced images from live image data (e.g., parameters of a function used to convert or correct live image data into enhanced images). Thus, in some embodiments, the visual subsystem 124 may generate corrected video stream data to be displayed to the user based on (i) video stream data representing the user's environment and (ii) correction profiles associated with the user.

[0219] As an example, by performing a visual examination, it may be determined whether or not a deviation in the user's eyes exists, the deviation in the user's eyes may be measured, or one or more correction profiles for the user's eyes may be generated. In one use case, with respect to Figure 25A, when the target stimulus 2502 is presented to the patient (e.g., a patient without strabismus) at a central position on the right and left displays 2503a and 2503b of the wearable device, both eyes (e.g., the right and left eyes 2504a and 2504b) instinctively move to fixate on the target stimulus 2502 at a central position on each wearable display, and thus the patient will only see one target stimulus 2502. Thus, based on the eye response described above, the examination subsystem 122 may determine that the user does not have diplopia.

[0220] In another use case, with respect to Figure 25B, when a target stimulus 2502 is presented to a patient with strabismus at the center position of the left and right displays of a wearable device, one eye (e.g., the dominant eye) instinctively moves to the center position and fixates on the target stimulus 2502 (e.g., the left eye 2504b instinctively fixates on the target stimulus 2502). The other eye (e.g., the right eye 2504a) also moves, but because this eye has exotropia, it does not fixate on the target stimulus 2502, resulting in double vision for the user (e.g., the user sees two target stimuli instead of one). For example, the other eye moves instinctively, but this instinctive movement causes the gaze direction of the other eye to be directed to a different position. Based on the above eye responses, the examination subsystem 122 may determine that the user has diplopia. However, in a further use case, as the user focuses on viewing the target stimulus 2502 with the other eye (e.g., the right eye 2504a with strabismus), the other eye shifts and fixates on the target stimulus 2502 presented in the center position on the user interface for the other eye. Since the target stimulus 2502 is presented in the center position on both displays 2503a and 2503b, the dominant eye remains dominant and continues to fixate on the target stimulus 2502 presented in the center position on the dominant eye's display. To determine the deviation measurement of the other eye, corrective movements of the other eye (and other changes in the eye-related properties of the other eye) may be measured (e.g., the amount of movement of the other eye may correspond to the amount of strabismus of the other eye).

[0221] In another use case, with respect to Figure 25C, at time t1, the stimulus (e.g., target stimulus 2502) may be presented to the left eye 2504b only in the central position by presenting the stimulus on the left display 2503b and not on the right display 2503a. For example, if the stimulus is presented to both eyes 2504a and 2504b in the central position immediately before presenting the stimulus to the left eye 2504b only (e.g., at time t0 immediately before the stimulus presentation at time t1), as shown in Figure 25B, the left eye 2504b will not move because it is already fixated on the central position. However, if the left eye 2504b is not yet fixated on the central position, presenting the stimulus to the left eye 2504b only will cause the left eye 2504b to instinctively move to the central position and fixate on the target stimulus 2502.

[0222] As shown in Figure 25D, the stimulus (e.g., target stimulus 2502) may be presented to the right eye 2504a only (e.g., at time t2) at the central position by presenting the stimulus on the right display 2503a and not presenting the stimulus on the left display 2503b. Since the left eye 2504b is not stimulated (e.g., there is nothing to look at), the left eye 2504b loses its dominance, and the right eye 2504a becomes dominant, causing the left eye 2504b to move outward. If the target stimulus 2502 is presented to the right eye 2504a only, the right eye 2504a will instinctively become dominant and move to fixate on the central position. The examination subsystem 122 may measure the movement of the right eye 2504a to determine the deviation measurement of the right eye 2504a (e.g., the amount of movement may correspond to the amount of strabismus of the right eye 2504a).

[0223] As shown in Figure 25E, the stimulus (e.g., target stimulus 2502) may be presented to both eyes 2504a and 2504b at the central position (e.g., at time t3) by presenting the stimulus on the left display 2503b and the right display 2503a. If strabismus occurs alternately (neither eye is dominant), the right eye 2504a will remain fixed at the central position, while the left eye 2504b will remain strabismic. However, if the left eye 2504b is the dominant eye (as shown in Figure 25E), the left eye 2504b will instinctively move and fixate at the central position. This movement of the left eye 2504b causes the right eye 2504a to become strabismic, resulting in the right eye 2504a's gaze direction shifting to a different position. The inspection subsystem 122 may measure the movement of the left eye 2504b in order to determine or confirm the deviation measurement of the right eye 2504a (for example, the amount of movement of the left eye 2504b may correspond to the amount of deviation of the right eye 2504a).

[0224] In further use cases, further testing may be performed to confirm the deviation measurement of the non-dominant eye. For example, as shown in Figure 25F, following one or more of the aforementioned steps described with respect to Figures 25B-25E, the stimulus (e.g., target stimulus 2502) may be presented to the left eye 2504b only (e.g., at time t4) in a central position by presenting the stimulus on the left display 2503b and not on the right display 2503a. Insofar as the left eye 2504b loses fixation (e.g., due to the presentation of Figure 25E), the presentation of Figure 25F will cause the left eye 2504b to instinctively move to regain fixation in the central position. This movement of the left eye 2504b causes the right eye 2504a to become strabismic, resulting in the right eye 2504a's gaze direction being shifted to a different position. As shown in Figure 25G, a corrective position for presenting the stimulus to the right eye 2504a may be determined based on the deviation measurement of the right eye 2504a. Thus, while the target stimulus 2502 is presented at the central position on the left display 2503b, the target stimulus 2502 may also be presented at a modified position on the right display 2503a (for example, at time t5).

[0225] Subsequently, with respect to Figure 25H, the target stimulus 2502 may be presented only to the right eye 2504a (for example, at time t6) by presenting it in a corrected position on the right display 2503a and not on the left display 2503b. Specifically, for example, the target stimulus 2502 is deviated to the right by the same amount as the deviation measured in one or more of the aforementioned steps described with respect to Figures 25B-25E. If the deviation measurement is accurate, the right eye 2504a does not move. If the deviation measurement is inaccurate, the right eye 2504a moves slightly, and the amount of this movement may be measured by a wearable device (e.g., a pupil tracking device of a wearable device), and this measurement of slight movement may be used to fine-tune the deviation. As an example, the measured value and corrected position may be used to determine an updated corrected position for presenting the stimulus to the right eye 2504a, and one or more of the steps described with respect to Figures 25F-25H may be repeated using the updated corrected position. Furthermore, or alternatively, one or more of the steps in Figures 25B–25E may be repeated to re-determine the deviation measurement for one or more of the user's eyes (e.g., to re-determine the deviation measurement for the right eye 2504a). With respect to Figure 25I, the target stimulus 2502 may then be presented to both eyes 2504a and 2504b (e.g., at time t7) by presenting the target stimulus 2502 in front of the right eye 2504a to the right according to the deviation measurement (e.g., determined or confirmed in one or more of the steps described above), so that the user does not see double, thereby providing autonomous correction for the patient's diplopia.

[0226] In some embodiments, a visual examination may be performed to determine which of the user's eyes is deviated. Based on such a determination, the deviation of the deviated eye may be measured, and the deviation measurement may be used to generate a corrective profile to compensate for the user's visual deviation. As an example, the examination subsystem 122 may present stimuli at a first position on a first user interface for the first eye and at a first position on a second user interface for the second eye at a predetermined time. The examination subsystem 122 may detect that the first eye is not fixating on the first position when the stimuli are presented on the first user interface. Based on the detection that the first eye is not fixating, the examination subsystem 122 may determine that the user's first eye is deviated. As an example, with respect to Figure 25B, when a target stimulus 2502 is presented to a patient with strabismus at the center position of the left and right displays of a wearable device, one eye (e.g., the dominant eye) instinctively moves to the center position and fixates on the target stimulus 2502 (e.g., the left eye 2504b instinctively fixates on the target stimulus 2502). The other eye (e.g., the right eye 2504a) also moves, but because this eye has exotropia, it does not fixate on the target stimulus 2502, resulting in double vision for the user (e.g., the user sees two target stimuli instead of one). Based on the detection of this lack of fixation, it may be determined that the other eye is the deviated eye.

[0227] In some embodiments, a visual inspection may be performed while the eyes are looking in different directions of gaze, and the degree of diplopia in each direction of gaze may be detected. In this way, diagnosis and correction for a specific type of strabismus (e.g., non-interference strabismus) can be performed. For example, in a patient with paralyzed eye muscles, the deviation (strabismus angle) between the two eyes increases when looking in the direction of action of the muscles. For example, if the muscle that turns the left eye outward is paralyzed, the left eye will look inward (commonly known as esotropia). When the left eye tries to look outward, the degree of strabismus increases. This is a phenomenon that occurs in paralytic strabismus. The wearable device (or other components connected to the wearable device) can accurately measure the deviation angle by repeating the quantification test while presenting stimuli in different regions of the visual field. Also, by knowing the degree of deviation in different directions of gaze, dynamic correction of diplopia becomes possible. Such a visual inspection presentation is provided via the wearable device, and when the pupil tracking device of the wearable device detects that the eye is at a specific fixation point, the wearable device may displace an image corresponding to that fixation point.

[0228] In some embodiments, such an examination can be performed when the patient is looking at a distant object and when looking at a nearby object. In some embodiments, the wearable device can automatically examine the range of motion of the extraocular muscles by presenting moving stimuli. When the patient follows it with their eyes, the wearable device (or other components connected to the wearable device) measures the range of motion and determines information regarding the user's diplopia based on the measurement results of the range of motion.

[0229] Thus, in some embodiments, multiple correction profiles can be generated for a user to correct deficiencies in dynamic vision (e.g., diplopia or other vision deficiencies). As an example, a first correction profile associated with a user may include one or more correction parameters applied to correct an image for the user's first eye in response to the fixation direction of the second eye being directed to a first position, the second eye having a first torsion (e.g., a first torsion angle), or other characteristics of the second eye. A second correction profile associated with the user may include one or more correction parameters applied to correct the image for the first eye in response to the fixation direction of the second eye being directed to a second position, the second eye having a second torsion (e.g., a second torsion angle), or other characteristics of the second eye. A third correction profile associated with the user may include one or more correction parameters applied to correct the image for the first eye in response to the line-of-sight direction of the second eye being directed to a third position, the second eye having a third torsion (e.g., a third torsion angle), or other characteristics of the second eye, and so on. In one usage example, one or more of the steps described with respect to FIGS. 25B-25H may be repeated for one or more other positions (in addition to or instead of the central position) to generate multiple correction profiles for the user.

[0230] In some embodiments, the visual subsystem 124 may monitor one or more eye-related characteristics of one or more of the user's eyes and generate modified video stream data to be displayed to the user based on (i) video stream data representing the user's environment, (ii) monitored eye-related characteristics, and (iii) a correction profile associated with the user. For example, if monitoring indicates that the gaze direction of the second eye is directed towards a first position, the video stream data may be modified using a first correction profile (e.g., its correction parameters) to generate modified video stream data to be displayed to the user's first eye. In another example, if monitoring indicates that the gaze direction of the second eye is directed towards a second position, the video stream data may be modified using a second correction profile (e.g., its correction parameters) to generate modified video stream data for the user's first eye. The same applies hereafter. Thus, for example, the above description illustrates the typically dynamic nature of diplopia (e.g., diplopia increasing or decreasing towards one or more lines of sight). For example, if there is a problem moving the user's right pupil away from the user's nose (e.g., towards the edge of the user's face), the user's double vision may increase when the user is looking to the right and decrease when the user is looking to the left. In this way, appropriate corrections can be made by monitoring the user's pupil, gaze, or other eye-related characteristics and applying an appropriate correction profile specific to the user's real-time eye-related characteristics to the live video stream data.

[0231] In some embodiments, a visual examination may be performed to evaluate the user's binocular vision. In some embodiments, a wearable device may be used to perform the binocular vision examination. For example, one or more stimuli may be presented on the user interface of each wearable device display targeted at the user's eyes, where the number or type of stimuli presented on one user interface is different from the number or type of stimuli presented on the other user interface (for example, the number of stimuli on each user interface may be different, or at least one stimulus on one user interface may have a different color or pattern from the stimuli on the other user interface). Alternatively, in some scenarios, the number or type of stimuli presented on both user interfaces may be the same. The examination subsystem 122 may determine whether the user has diplopia based on the user's instructions regarding the number or type of stimuli the user is looking at.

[0232] In one use case, with respect to Figure 25J, the binocular vision test may include the user wearing a wearable device having displays 2522a and 2522b (or viewing these displays 2522a and 2522b via another device), where each display 2522 is configured to present one or more stimuli to each of the user's eyes, or to provide other presentations. For example, stimuli 2524a and 2524b (e.g., green dots) may be presented to one of the user's eyes on display 2522a, and stimuli 2526a, 2526b, and 2526c (e.g., red dots) may be presented to the user's other eye on display 2522b. With respect to Figure 25K, the test subsystem 122 may determine that the user is seeing binocular monopsy (and therefore not diplopia) based on the user's instruction that the user is looking at four dots. Furthermore, or alternatively, the testing subsystem 122 may determine or confirm that the user has binocular monopsy based on the user's instruction that the user is looking at one green dot (e.g., stimulus 2524a), two red dots (e.g., stimuli 2526a and 2526c), and one mixed-color dot (e.g., mixed stimulus 2528, a combination of stimuli 2524b and 2526b). On the other hand, with respect to Figure 25L, the testing subsystem 122 may determine that the user has double vision (e.g., diplopia) based on the user's instruction that the user is looking at five dots. Furthermore, or alternatively, the testing subsystem 122 may determine or confirm that the user has diplopia based on the user's instruction that the user is looking at two green dots (e.g., stimuli 2524a and 2524b) and three red dots (e.g., stimuli 2526a, 2526b, and 2526c).

[0233] In some embodiments, the examination subsystem 122 may monitor one or more eye-related characteristics related to the user's eyes during a visual examination presentation via two or more user interfaces (e.g., on two or more displays) and autonomously determine whether the user has diplopia based on the eye-related characteristics occurring during the visual examination presentation. In some embodiments, the examination subsystem 122 may determine the degree of the user's diplopia based on such eye-related characteristics (e.g., by measuring the deviation of one or more eyes as described herein) and autonomously generate one or more correction profiles to correct the diplopia. As an example, the wearable device may include pupil and eye-tracking devices for detecting the gaze direction or other eye-related characteristics of one or more of the user's eyes. Based on the gaze direction (or other eye-related characteristics), the examination subsystem 122 may determine the number of points the user has fixed on (this is done, for example, by using the detected gaze direction to determine whether the user fixed on a location corresponding to a presented stimulus). In one use case, with respect to Figure 25J, if it is determined that the user has fixed on four points (for example, points corresponding to stimuli 2524a, 2526a, 2526c, and 2528 shown in Figure 25K), the testing subsystem 122 may determine that the user does not have diplopia. If it is determined that the user has fixed on five points (for example, points corresponding to stimuli 2524a, 2524b, 2526a, 2526b, and 2526c shown in Figure 25L), the testing subsystem 122 may determine that the user has diplopia.

[0234] As a further example, in response to determining that the user has fixed on a particular point (e.g., a point corresponding to the display position of multiple presented stimuli or each of them), the examination subsystem 122 may mitigate the effect of the corresponding stimulus and increase the count of the number of stimuli the user is looking at. For example, the corresponding stimulus may be removed from the presentation of the visual examination (e.g., the corresponding stimulus may disappear and the remaining stimuli may continue to be presented), or it may be modified to reduce its effect (this is done, for example, by reducing the brightness or other intensity level of the stimulus). As another example, the relative effect of the corresponding stimulus can be reduced by modifying the other stimuli to increase their effect (e.g., by increasing the brightness or other intensity level of the other stimuli). In this way, the user's eyes instinctively move to and fixate on one or more points corresponding to the remaining stimuli. With respect to Figure 25K, for example, stimuli 2524b and 2526b (represented by the mixed stimulus 2528) are removed when the user's eyes fixate on the positions corresponding to stimuli 2524b and 2526b. On the other hand, in the case of Figure 25L (where the user has diplopia), stimuli 2524b and 2526b are removed at two different times because the user does not fixate on the same relative position when looking at stimulus 2524b or 2526b. The testing subsystem 122 may continue to remove stimuli each time the user fixates on the corresponding point and increment the count (of the number of stimuli the user sees). When all stimuli have been removed, or when other thresholds are met, the testing subsystem 122 may provide the number of stimuli the user sees.

[0235] In some embodiments, the examination subsystem 122 may determine whether or not the user has stereopsis, or the degree of the user's stereopsis, based on eye-related characteristics occurring during the visual examination presentation. For example, the examination subsystem 122 may cause one or more stimuli to be presented at one or more locations on one or more user interfaces and autonomously perform such a determination of stereopsis or other visual deficiency information based on eye-related characteristics. In one use case, with respect to Figure 25M, the visual examination presentation may include the user wearing a wearable device having displays 2542a and 2542b (or viewing these displays 2542a and 2542b via another device), each display 2542 being configured to present one or more stimuli or to provide other presentations to the user's respective eyes.

[0236] As shown in Figure 25M, one or more icons 2544 or other stimuli may be presented on each display 2542, where one or more pairs of icons 2544 are presented in corresponding positions on both displays 2542, and at least one pair of icons 2544 are presented in slightly different positions on displays 2542a and 2542b. In particular, in Figure 25M, the arrangement of icons 2544 on both displays 2542 is the same, except that the icon 2544 in the second row and third column on display 2542b is slightly shifted upward and to the right (as indicated by indicator 2546). For users without binocular diplopia or stereopsis, this slight difference causes the pair of icons to appear as three-dimensional icons, while all other icons 2544 appear as two-dimensional icons. Therefore, the user instinctively shifts their gaze to the three-dimensional icons and fixates on them. Based on the determination that an individual has fixed on a three-dimensional icon (for example, within a predetermined threshold time), the testing subsystem 122 may determine that the user does not have stereopsis. For example, the testing subsystem 122 may detect that the gaze direction of one or more of the user's eyes changes when a stimulus is presented, and that the gaze direction is now directed towards the area on which the corresponding icon 2544 is presented on each display 2542.

[0237] However, if the user has stereoscopic vision, slight differences may cause the icon pair to not appear as a three-dimensional icon to the user, and the user is likely not to fixate on the corresponding area where the icon pair is presented on each display 2542. Based on this failure to fixate (for example, within a predetermined threshold time), the inspection subsystem 122 may determine that the user has stereoscopic vision.

[0238] In a further use case, with respect to Figure 25M, the amount of difference between two icons 2544 in the second row, third column may be modified to determine the degree of the user's stereoscopic vision. For example, icon 2544 (within the area indicated by indicator 2546) may be initially shifted up or to the right so that the difference in position between icon 2544 on display 2542b and the corresponding icon 2544 on display 2542a is minimized. If the user does not fixate on the corresponding area where the pair of icons is presented, icon 2544 on display 2542b may be shifted up or to the right again so that the difference in position between the two icons 2544 is slightly increased. The positional difference may be repeatedly increased until the user fixates on the corresponding area or until a threshold of positional difference is reached. The inspection subsystem 122 may use the amount of positional difference (or the number of times the shift operation has been performed) to measure the degree of the user's stereoscopic vision.

[0239] In another use case, with respect to Figure 25N, the stimulus presentation during the visual test presentation may be provided in the form of randomly generated noise. In Figure 25N, the stimulus presented on display 2562a and the stimulus presented on display 2562b are identical except that the set of blocks (e.g., pixels) in the area indicated by indicator 2564 is shifted 5 units (e.g., pixels) to the right on display 2562b (compared to the same set of blocks on display 2562a). Similar to the aforementioned use case with respect to Figure 25M, this slight difference causes the set of blocks to appear (or stand out) as a three-dimensional object to a user without binocular diplopia and stereopsis, resulting in the user quickly fixing on the three-dimensional object. Based on the determination that the individual has fixed on the three-dimensional object, the test subsystem 122 may determine that the user does not have stereopsis. However, if the user does have stereopsis, the slight difference may cause the user to not notice the set of blocks, and the user will not fixate on the corresponding area where the set of blocks is presented on each display 2562. Based on the absence of fixation, the inspection subsystem 122 may determine that the user has stereoscopic vision.

[0240] In some embodiments, with respect to Figure 1A, the visual subsystem 124 may facilitate the expansion of the user's field of vision through the combination of multiple image portions of a scene (for example, based on feedback related to a set of stimuli displayed to the user). As an example, Figure 26 shows a normal binocular vision of a subject. In this case, a monocular image from the left eye 2602 and a monocular image from the right eye 2604 are combined to form a single perceived image 2606 that includes the central macular region 2608 and the peripheral visual field region 2610 surrounding the central region 2608. However, in some cases, the subject may have symptoms of constricted visual field, in which case the peripheral region 2610 is not visible to the subject, as shown in Figure 27. As illustrated, in such cases, one or more objects do not appear in the field of vision, a peripheral defect 2612 is observed in region 2610, and objects within region 2610 are not visible to the subject. Therefore, for example, the visual subsystem 124 may expand the subject's field of vision by combining portions of multiple images of a scene (for example, common and differing regions of such images).

[0241] In some embodiments, the visual subsystem 124 may acquire multiple images of a scene (for example, images obtained through one or more cameras at different positions or orientations). The visual subsystem 124 may determine regions common to the multiple images and, for each of the multiple images, determine regions of the image that are different from corresponding regions in at least one other image among the multiple images. In some embodiments, the visual subsystem 124 may generate or display an enhanced image to the user based on the common and different regions. For example, the common and different regions may be combined to generate an enhanced image that includes representations of the common and different regions. The common regions may correspond to each part of a group of images that have the same or similar characteristics as each other, and each different region may correspond to a part of an image that is different from all the corresponding parts of the other images. In a scenario, the part of one image that is different from the others may include a part of the scene that is not represented in the other images. In this way, for example, by combining the common and different regions to create an enhanced image, the field of view provided for each image otherwise is expanded, and the user's field of view can be enhanced using the enhanced image. In one use case, the common region may be any portion between any two of the four vertical dotted lines shown in Figure 27 for each of the images, at least one of the left eye image 2602 or the right eye image 2604. In another use case, with respect to Figure 27, one of the difference regions may be any portion to the left of the leftmost vertical dotted line in the left eye image 2602. Another difference region may be any portion to the right of the rightmost vertical dotted line in the right eye image 2604.

[0242] In some embodiments, the common region is a region in at least one of the images corresponding to the macular region (or other central region of the eye's visual field) or a region within the macular region of the eye's visual field. In some embodiments, each of the difference regions is a region in at least one of the images corresponding to the peripheral region of the eye's visual field or a region within the peripheral region. As an example, with respect to Figure 27, the common region may be (i) a portion of the image corresponding to the macular region of the left eye 2602, or (ii) a portion of the image corresponding to the macular region of the right eye 2604 (assuming, for example, that these portions are common to both images). As another example, the common region may be a portion of the image corresponding to a common region within the macular region of the left eye 2602 and the right eye 2604. As a further example, based on the common and difference regions, image 2606 is generated to have a central macular region 2608 and a peripheral visual field region 2610 surrounding the central region 2608.

[0243] In some embodiments, the visual subsystem 124 may determine a region common to multiple images of a scene (e.g., captured via the user's wearable device), and for each of these images, determine a region of the image that is different from a corresponding region in at least one other image among these images. The visual subsystem 124 may perform a shift on each of these images, and following the shift, generate an enhanced image based on the common and differing regions. In some embodiments, the shift on each image may be performed such that (i) the size of the common region is modified (e.g., increased or decreased), or (ii) the size of at least one of the differing regions is modified (e.g., increased or decreased). In one scenario, the size of the common region may increase as a result of the shift. In another scenario, the size of at least one of the differing regions may decrease as a result of the shift.

[0244] As an example, the loss shown in Figure 27 may be corrected using shift image correction technology. In one use case, with respect to Figure 28, each of the two field-of-view cameras (e.g., of a wearable device) may capture monocular images 2802 and 2804 respectively (for example, each monocular image is different because it captures a scene seen from a slightly different (shifted) position). Then, the two captured monocular images 2802 and 2804 are shifted closer to each other by a visual correction framework to obtain images 2802' and 2804'. As shown in Figure 28, the respective regions of the two images 2802 and 2804 between the leftmost and rightmost vertical dotted lines of each image 2802 and 2804 (e.g., the common region) are the same as the leftmost and rightmost vertical dotted lines of each image 2802' and 2804'. The respective regions of the two images 2802' and 2804' between the dotted line (for example, the common region) ) is larger than . Thus, the common region decreases in size after the shift. On the other hand, the difference region increases in size after the shift (for example, comparing the region to the left of the leftmost vertical dotted line in image 2802 with the region to the left of the leftmost vertical dotted line in image 2802', The area to the right of the rightmost vertical dotted line in Image 2804 and the rightmost vertical dotted line in Image 2804' (Compare this with the area on the right.)

[0245] As a further example, these two shift images are then combined to generate a binocular image 2806 that captures the entire peripheral area of ​​the visible scene. For eyeglasses with monitor displays, each display may show the modified binocular image 2806 to the subject. In some use cases, for example, this shift transformation can expand the subject's field of view by 5%, 10%, 15%, 20%, or more without causing double vision.

[0246] In some embodiments, the visual subsystem 124 may determine a region common to multiple images of a scene (e.g., captured via the user's wearable device), and for each of these images, it may determine a region of the image that is different from a corresponding region in at least one other image among these images. The visual subsystem 124 may resize one or more regions of these images and, following the resizing, generate an enhanced image based on the common and differing regions. In some embodiments, when the visual subsystem 124 resizes one or more regions of the images, it may ensure that any range of resizing of the common region is different from any range of resizing of at least one of the differing regions. In some embodiments, when resizing is performed, the rate of change of the size of the common region represented in a first region of the enhanced image may be greater than or less than the rate of change of the size of at least one of the differing regions represented in a second region of the enhanced image. For example, the rate of change of the size of at least one of the differing regions may be zero, and the rate of change of the size of the common region may be greater than zero. As another example, the rate of change of the size of at least one of the differing regions may be greater than zero, and the rate of change of the size of the common region may be zero.

[0247] In one scenario, with respect to Figure 29, the captured monocular images 2902 and 2904 are resized, for example, only in the peripheral region, while the central macular region (the central 20 degrees) remains unchanged, generating modified images 2902' and 2904'. This resizing transformation expands the field of view while maintaining visual acuity in the center. As shown in Figure 29, the combined binocular image 2906 captures peripheral objects that were previously invisible, while simultaneously maintaining the detail of the central macular region. Peripheral objects are clearly perceived by the subject even after resizing. This is because peripheral visual acuity is not as sensitive as central visual acuity. In some use cases, for example, a reduction of up to 20% in image size can be performed without causing diplopia in the subject. In various embodiments, resizing of the peripheral region may be performed in addition to, or instead of, resizing of the central region. For example, the peripheral region may be resized to the size of the peripheral region while maintaining the size of the central macular region (e.g., in the case of glaucoma patients). In another scenario, for patients with macular degeneration, the peripheral vision may be left unchanged (e.g., without resizing), while the central region may be resized to reduce its size. An enhanced image (e.g., a binocular image) can then be generated to include the resized central region.

[0248] In some embodiments, the visual subsystem 124 may determine a region common to multiple images of a scene (e.g., those captured via the user's wearable device), and for each of these images, determine a region of the image that is different from a corresponding region in at least one other image among these images. The visual subsystem 124 may perform a fisheye transformation, conformal transformation, or other transformation on the common region, and subsequently generate an enhanced image based on the common and differing regions. In some embodiments, the visual subsystem 124 may perform a fisheye transformation, conformal transformation, or other transformation on a region of the enhanced image (including the common region).

[0249] As an example, according to the following formula, a fisheye transformation can be performed on a region to modify the radical component of an image. r new =r + αr 3 Here, α is a constant.

[0250] As another example, according to the following formula, a conformal transformation can be performed on a region to modify the radius component of an image. r new =rβ Here, β is a constant exponent for the radial component and β > 1.

[0251] In some embodiments, the visual subsystem 124 may modify at least one of the plurality of images of the scene by moving one or more objects within the image (e.g., before generating an enhanced image based on common and difference regions of the image). As an example, with respect to FIG. 30, for a patient with a significant peripheral defect in one eye, an object 3002 that is not visible in the visual field 3004 of the eye with the defect can be digitally moved to the mid-peripheral region 3006 of the visual field 3004, while the visual field 3008 of the other normal eye covers this region. That is, the object 3002 that was not visible is displayed within the normal visual field in the combined binocular image 3010. The subject will notice visual confusion in this region, but can adapt to separate the information in this region of the visual field according to the moving object or changing environment.

[0252] In some embodiments, the visual subsystem 124 may determine one or more defective fields of view of the user's field of view (for example, according to one or more techniques described herein). In some embodiments, the visual subsystem 124 may determine a region common to multiple images of a scene (for example, captured via the user's wearable device), and for each of these images, determine a region of the image that is different from a corresponding region in at least one other image among these images. The visual subsystem may generate an enhanced image based on the common and differing regions of the images such that at least one of the common or differing regions in the enhanced image does not overlap with one or more defective fields of view.

[0253] In some embodiments, the visual subsystem 124 may detect objects in the defective portion of the user's field of vision and display an alert. For example, after correcting the defective portion of the user's field of vision (e.g., via one or more techniques described herein), the visual subsystem 124 may monitor the remaining uncorrected area to detect one or more objects (e.g., safety issues or other objects) and generate an alert (e.g., a visual or auditory alert) indicating these objects, their location, size, or other information related to the objects. In one use case, for a patient with an irregular or multi-regional defective field of vision, the generated correction profile may still not be optimal in fitting the acquired field of vision to the normal area of ​​the patient's field of vision. Therefore, to maximize the safety of the moving patient, an automated video tracking algorithm may be implemented to detect objects in one of the defective portions of the field of vision. Such objects may include moving objects (e.g., a moving car) or other objects in the defective portion of the patient's field of vision.

[0254] In some embodiments, the visual subsystem 124 may generate predictions indicating that an object will physically contact the user and may display an alert based on this physical contact prediction (e.g., an alert related to the object is displayed on the user's wearable device). In some embodiments, the visual subsystem 124 may detect an object (e.g., in or predicted to be in a defective portion of the user's field of vision) and may display an alert based on (i) that the object is in or predicted to be in a defective portion of the field of vision, (ii) a physical contact prediction, or (iii) other information. In some embodiments, the visual subsystem 124 may determine whether an object is outside (or not fully in) any portion of the enhanced image (displayed to the user) corresponding to at least one portion of the field of vision that satisfies one or more visual criteria. In one use case, if it is determined that an object is within (or fully in) the portion of the enhanced image corresponding to a healthy portion of the user's field of vision, no alert may be displayed (or a lower-priority alert may be displayed) (e.g., even if the object is predicted to physically contact the user). On the other hand, if an object within a defective field of view is predicted to make physical contact with the user, and it is determined that the object is outside (or not sufficiently within) the user's healthy field of view, an alert may be displayed on the user's wearable device. In this way, for example, the user can rely on their own healthy field of view to avoid objects approaching within their healthy field of view, thereby reducing the risk of relying on the wearable device (e.g., through habit formation) to avoid such approaching objects. However, it should be noted that in other use cases, an alert related to the object may be displayed based on the prediction of physical contact, regardless of whether the object is within the user's healthy field of view.

[0255] As an example, with respect to Figure 10, for the uncompensated blind spot field 1006, in blocks 1012 and 1014, pupil tracking or other visual tracking (e.g., using an inwardly oriented image sensor) or video tracking of a moving object in the field of view (e.g., using an outwardly oriented image sensor such as an external camera) may be used to detect a safety issue that is in or moving into the blind spot area. In one use case, the visual subsystem 124 may compare the location of a safety issue (e.g., measured in inspection mode) to a defective mapped field of view to detect when the safety issue is in or moving into the blind spot area.

[0256] As another example, after correcting a defective portion of the user's field of vision (e.g., via one or more techniques described herein), the visual subsystem 124 may monitor the remaining uncorrected areas to detect (e.g., in real time) any safety issues approaching the user from such areas. If such detected safety issues are expected to make physical contact with the user or enter within the user's threshold distance (e.g., one foot, two feet, or other threshold distances) (as opposed to passing near the user at or above the user's threshold distance), the visual subsystem 124 may generate an alert related to the detected safety issue (e.g., a visual alert displayed in an area visible to the user, an auditory alert, etc.).

[0257] In one use case, video signals (e.g., live video streams) acquired from one or more cameras on the user's wearable device are preprocessed and filtered to remove residual noise effects. In another use case, the search area may be limited to the user's blind spots or other defective areas of the field of view (e.g., those that do not satisfy one or more visual criteria). Limiting the search area can reduce the amount of computational resources required to detect objects within the search area or generate related alerts, or it can increase the speed of such detection and alert generation.

[0258] In some cases, the movement of one or more objects may be detected by subtracting two consecutive frames from a live video stream from each other. For example, the occurrence of motion may be stored in a first delta frame (e.g., delta frame 1), which may be used to enable visualization of the moving object and to offset a stationary background. A second delta frame (e.g., delta frame 2) may be generated by subtracting two other consecutive frames from the live video stream from each other. The second delta frame may also be used to enable visualization of the moving object and to offset a stationary background. Furthermore, a comparison may be made between the first and second delta frames. If it is detected that the size of a moving object is increasing by subtracting the first and second delta frames from each other, it may be determined that the object is approaching. If the size increase exceeds a predetermined threshold size, an alert is issued to the user (e.g., a visual alert displayed in an area visible to the user, an audible alert, etc.).

[0259] In some embodiments, the configuration subsystem 112 may store predictive models, correction profiles, visual defect information (e.g., indicating a detected user's visual defect), feedback information (e.g., feedback related to stimuli displayed to the user, or other feedback), or other information in one or more remote databases (e.g., the cloud). In some embodiments, feedback information, visual defect information, correction profiles, or other information related to multiple users (e.g., two or more users, ten or more users, one hundred or more users, one thousand or more users, one million or more users, or any other number of users) may be used to train one or more predictive models. In one use case, if the predictive model being trained is a neural network or other machine learning model, the model manager subsystem 114 may provide (i) and (ii) below as inputs to the machine learning model to cause the machine learning model to predict visual defect information, correction profiles, or other outputs: (i) stimulus information (e.g., a set of stimuli and their associated characteristics, such as intensity levels and locations where stimuli are displayed), and (ii) feedback information (e.g., feedback related to a set of stimuli). The model manager subsystem 114 may provide the machine learning model with reference information (e.g., visual defect information or correction profiles that are deemed accurate with respect to the provided stimulus and feedback information). The machine learning model may evaluate the predicted output (e.g., predicted visual defect information, predicted correction profiles, etc.) in comparison to the reference information and update its configuration (e.g., weights, biases, or other parameters) based on its evaluation of the predicted output.The aforementioned operations can be performed with additional stimulus information (e.g., what is displayed to other users), additional feedback information (e.g., feedback from other users related to stimuli displayed to other users), and additional reference information to further train the machine learning model (for example, the machine learning model can further update its configuration by providing such information as input and reference feedback in order to train the machine learning model).

[0260] In another use case, if the machine learning model is a neural network, the connection weights may be adjusted to adjust for the difference between the neural network's predictions and the reference information. In a further use case, one or more neurons (or nodes) in the neural network may request that their respective errors be sent backward through the neural network to facilitate an update process (e.g., backpropagation). The update of the connection weights may reflect, for example, the magnitude of the error to be propagated backward after forward propagation is complete.

[0261] In some embodiments, one or more predictive models may be trained or configured for a user or a predetermined type of device (e.g., a device of a specific brand, a device of a specific brand and model, a device having a specific set of features, etc.) and may be stored in association with the user or device type. For example, instances of a predictive model associated with a user or device type may be stored locally (e.g., on the user's wearable device or other user devices) and remotely (e.g., in the cloud), and such instances of the predictive model may be automatically or manually synchronized between one or more user devices and the cloud so that the user can access the latest configuration of the predictive model on either the user device or the cloud. In one use case, when it is detected that a first user is using a wearable device (e.g., when the first user logs into their account or is identified through one or more other technologies), the configuration subsystem 112 may communicate with the wearable device to send the latest instance of the predictive model associated with the first user to the wearable device so that the wearable device can access a local copy of the predictive model associated with the first user. In another use case, if it is later detected that a second user is using the same wearable device, the configuration subsystem 112 may communicate with the wearable device to send the latest instance of the predictive model associated with the second user to the wearable device, allowing the wearable device to access a local copy of the predictive model associated with the second user.

[0262] In some embodiments, multiple correction profiles may be associated with a user or device type. In some embodiments, each correction profile may include a set of correction parameters or functions applied to live image data in a given context to generate an enhanced presentation of the live image data. For example, a user may have correction profiles for each set of eye characteristics (e.g., range of gaze direction, pupil size, margin position, or other characteristics). As a further example, a user may additionally or alternatively have correction profiles for each set of environmental characteristics (e.g., range of ambient brightness levels, ambient temperature, or other characteristics). Based on the currently detected eye or environmental characteristics, a corresponding set of correction parameters or functions may be obtained and used to generate an enhanced presentation of the live image data. In one use case, when it is detected that a first user is using a wearable device (e.g., when the first user logs into their account or is identified through one or more other technologies), the configuration subsystem 112 may communicate with the wearable device to send the correction profile associated with the first user to the wearable device, allowing the wearable device to access a local copy of the correction profile associated with the first user. In another use case, if it is later detected that a second user is using the same wearable device, the configuration subsystem 112 may communicate with the wearable device to send the correction profile associated with the second user to the wearable device, allowing the wearable device to access a local copy of the correction profile associated with the second user.

[0263] Figures 41 to 43 are examples of flowcharts illustrating the processing operations in methods for enabling the various features and functions of the system described in detail above. The processing operations of each method described below are illustrative and not limiting. In some embodiments, for example, these methods may be implemented by adding one or more processes not described, and / or by omitting one or more of the processes described. Furthermore, the order in which the processing operations of each method are illustrated (and described below) is not intended to be limiting.

[0264] In some embodiments, each method may be implemented in one or more processing devices (e.g., digital processors, analog processors, digital circuits for information processing, analog circuits for information processing, state machines, and / or other mechanisms that process information electronically). These processing devices may include one or more devices that perform some or all of the processing of each method in response to instructions electronically stored in an electronic storage medium. These processing devices may include one or more devices consisting of hardware, firmware, and / or software that are specifically designed to perform one or more of the processing of each method.

[0265] Figure 41 shows a flowchart of a method 4100 that facilitates user visual correction via a predictive model, according to one or more embodiments.

[0266] In step 4102, a visual test presentation may be provided to the user. For example, the visual test presentation may include a set of stimuli. The set of stimuli may include light stimuli, text, or images displayed to the user. According to one or more embodiments, step 4102 may be performed by a subsystem identical or similar to the test subsystem 122.

[0267] In step 4104, one or more characteristics of one or more of the user's eyes may be monitored. For example, eye characteristics may be monitored during the presentation of a visual test. Eye characteristics may include (e.g.) gaze direction, pupil size, periphery position, visual axis, optical axis, or other characteristics. According to one or more embodiments, step 4104 may be performed by a subsystem identical or similar to the test subsystem 122.

[0268] In step 4106, feedback related to the set of stimuli may be obtained. For example, the feedback may be obtained during the visual test presentation and may indicate whether or how the user is looking at one or more stimuli from the set. Furthermore, or instead, the feedback may include one or more characteristics related to one or more eyes that occur when one or more stimuli are displayed. According to one or more embodiments, step 4106 may be performed by a subsystem identical or similar to the test subsystem 122.

[0269] In step 4108, feedback related to the set of stimuli may be provided to the predictive model. For example, the feedback may be provided to the predictive model during the visual test presentation, and the predictive model may be configured based on the feedback and ocular characteristic information. As another example, based on the feedback, the predictive model may provide modification parameters or functions that are applied to the image data (e.g., a live video stream) to generate an enhanced presentation related to the image data. According to one or more embodiments, step 4108 may be performed by a subsystem identical or similar to the test subsystem 122.

[0270] In step 4110, video stream data and user's current eye characteristic information (e.g., information indicating the user's current eye characteristics) may be provided to the predictive model. For example, the video stream data may be a live video stream obtained through one or more cameras of the user's wearable device, and the live video stream and current eye characteristic information may be provided to the predictive model in real time. According to one or more embodiments, step 4110 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0271] In step 4112, a set of modification parameters or functions may be obtained from a predictive model. For example, the set of modification parameters or functions may be obtained from the predictive model based on the video stream and current eye characteristic information provided to the predictive model. As another example, the set of modification parameters or functions may be configured to be applied to the video stream to generate an enhanced image (e.g., an image that adapts to the user's dynamic aberrations). Furthermore, or alternatively, the set of modification parameters or functions may be configured to be applied to dynamically adjust one or more display portions of the display. According to one or more embodiments, step 4112 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0272] In step 4114, an enhanced image may be displayed to the user based on the video stream data and a set of modification parameters or functions. According to one or more embodiments, step 4114 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0273] Figure 42 shows a flowchart of a method 4200 that facilitates expanding the user's field of view through a combination of parts of multiple images of a scene, according to one or more embodiments.

[0274] In step 4202, multiple images of the scene may be acquired. For example, the multiple images may be acquired via one or more cameras (e.g., cameras of a wearable device) located at different positions or orientations. According to one or more embodiments, step 4202 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0275] In step 4204, a region common to multiple images may be determined. For example, the common region may correspond to each part of the images having the same or similar characteristics as each other. According to one or more embodiments, step 4204 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0276] In step 4206, for each of the multiple images, a region of the image that is different from a corresponding region of at least one other image (among the multiple images) may be determined. For example, each difference region may correspond to a portion of one of the multiple images, and this portion may be distinguished from the corresponding portions of all the other images. According to one or more embodiments, step 4206 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0277] In step 4208, an enhanced image may be generated based on common and differing regions. For example, the enhanced image may be generated such that (i) a first region of the enhanced image includes a representation of the common region, and (ii) a second region of the enhanced image includes a representation of the differing region. Alternatively, the enhanced image may be generated such that the second region surrounds the first region. According to one or more embodiments, step 4208 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0278] In step 4210, an enhanced image may be displayed. For example, the enhanced image may be displayed via one or more displays on the user's wearable device. According to one or more embodiments, step 4210 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0279] Figure 43 shows a flowchart of a method 4300 that facilitates the enhancement of a user's field of view through one or more dynamic display portions on one or more transparent displays, according to one or more embodiments.

[0280] Step 4302 may monitor one or more changes related to one or more of the user's eyes. For example, eye changes may include eye movements, changes in gaze direction, changes in pupil size, or other changes. According to one or more embodiments, step 4302 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0281] In step 4304, one or more transparent display portions of the wearable device may be adjusted based on the monitored changes. For example, the position, shape, or size of one or more transparent display portions of the wearable device may be adjusted based on the monitored changes. According to one or more embodiments, step 4304 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0282] In step 4306, the enhanced image (e.g., an image obtained from live image data) may be displayed on one or more other display portions of the wearable device. For example, at least one of the other display portions may be positioned around at least one of the transparent display portions of the wearable device so that the enhanced image is displayed around the transparent display portion (e.g., not inside the transparent display portion). According to one or more embodiments, step 4306 may be performed by a subsystem identical or similar to the visual subsystem 124.

[0283] In some embodiments, the various computers and subsystems illustrated in Figure 1A may include one or more computing devices programmed to perform the functions described herein. A computing device may include one or more electronic storage (e.g., one or more prediction databases 132, which may include one or more training data databases 134, one or more model databases 136, etc., or other electrical storage), one or more physical processors programmed with one or more computer program instructions, and / or other components. A computing device may include communication lines or communication ports that enable the exchange of information with a network (e.g., network 150) or other computing platform via wired or wireless technology (e.g., Ethernet, optical fiber, coaxial cable, WiFi, Bluetooth, near-field communication, or other technologies). A computing device may include multiple cooperating hardware components, software components, and / or firmware components. For example, a computing device may be implemented by multiple computing platforms cooperating as a computing device.

[0284] Electronic storage may include non-temporary storage media that electronically store information. The electronic storage media of electronic storage may include (i) system storage provided integrally with a server or client device (e.g., substantially indistinguishable), or (ii) removable storage detachably connected to a server or client device via, for example, a port (e.g., a USB port, a FireWire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage may include one or more of the following: optically readable storage media (e.g., optical discs, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drives, floppy drives, etc.), electrostatic storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drives, etc.), and / or other electronically readable storage media. Electronic storage may also include one or more virtual storage resources (e.g., cloud storage, virtual private networks, and / or other virtual storage resources). Electronic storage may store software algorithms, information determined by the processor, information obtained from a server, information obtained from a client device, or other information that enables the functions described herein.

[0285] A processor may be programmed to implement information processing functions in a computing device. For this purpose, a processor may include one or more of the following: a digital processor, an analog processor, digital circuits for information processing, analog circuits for information processing, a state machine, and / or other mechanisms for electronically processing information. In some embodiments, a processor may include multiple processing units. These processing units may be physically located within the same device, or multiple processors may perform processing functions for multiple devices operating in coordination. A processor may be programmed to execute computer program instructions to implement the functions of subsystems 112-124 or other subsystems described herein. A processor may be programmed to execute computer program instructions by software, hardware, firmware, a combination of software, hardware, or firmware, and / or other mechanisms for configuring processing functions in the processor.

[0286] The descriptions of the functions provided by the various different subsystems 112-124 described herein are illustrative and not intended to limit you. Each subsystem 112-124 may provide more or fewer functions than those described. For example, one or more subsystems 112-124 may be omitted, and some or all of their functions may be provided by other subsystems within subsystem 112-124. As another example, an additional subsystem may be programmed to perform some or all of the functions attributed to one of subsystems 112-124 herein.

[0287] The technology described herein may be used in any number of applications, for example, by healthy subjects, military personnel, and veterans who frequently experience instantaneous onset of optical-related disorders. Visual field defects reduce essential abilities for military personnel, veterans, and other patients, as well as their ability to perform daily activities. Such visual impairments reduce independence, safety, productivity, and quality of life, leading to decreased self-esteem and depression. Despite recent scientific advances, treatment options for restoring damage to the retina, optic nerve, or visual cortex are limited. Therefore, treatment mostly involves providing patients with visual aids to maximize function. Current visual aids are insufficient to achieve this goal. Therefore, there is a growing need for better visual aids to improve visual function, quality of life, and safety. The technology described herein, incorporated into eyeglasses, can diagnose and mitigate common eye injuries and disorders with instantaneous onset that cause visual field defects in harsh or isolated environments as well as in general environments, such as military-related eye injuries and disorders. The technology described herein makes it possible to diagnose and quantify visual field defects. The device uses this data to process the patient's visual field in real time and project a corrected image that fits the remaining functional visual field. This minimizes the adverse effects of blind spots (or areas of reduced visual acuity) on the patient's visual function. Furthermore, because the spectacle device does not require the use of another medical device to diagnose visual field defects, its usefulness is particularly enhanced in harsh or isolated environments. Similarly, the technology described herein may be used to enhance the visual field of a normal subject to achieve a better-than-normal visual field or visual acuity.

[0288] The present invention has been described in detail for illustrative purposes based on the most practical and preferred embodiments at present; however, it should be understood that such detailed descriptions are for illustrative purposes only. The present invention is not limited to the disclosed embodiments, and rather, variations and equivalent configurations that fall within the spirit and scope of the appended claims are also intended to be included in the present invention. For example, it should be understood that in the present invention, one or more features of any embodiment can be combined with one or more features of any other embodiment as much as possible.

[0289] The technology of the present invention will be better understood by referring to the embodiments listed below. [A1] A method comprising the steps of: providing a user with a presentation (e.g., a visual test presentation or other presentation) that includes a set of stimuli; obtaining feedback on the set of stimuli (e.g., the feedback indicating whether or how the user perceives one or more stimuli from the set); and providing the feedback on the set of stimuli to a model (e.g., a machine learning model or other model), the model being configured based on the feedback on the set of stimuli. [A2] A method of Embodiment A1, further comprising the steps of: providing live image data, eye characteristic information, or environmental characteristic information to the model to obtain an enhanced image obtained from the live image data; and displaying the enhanced image to the user, wherein the eye characteristic information indicates one or more characteristics of one or more of the user's eyes that occurred during the live imaging of the live image data, and the environmental characteristic information indicates one or more characteristics of the environment that occurred during the live imaging of the live image data. [A3] The method of Embodiment A2, further comprising the step of acquiring the enhanced image from the model based on the live image data, eye characteristic information, or environmental characteristic information provided to the model. [A4] A method of Embodiment A2, further comprising the steps of: obtaining one or more modification parameters from the model based on the live image data, eye characteristic information, or environmental characteristic information provided to the model; and generating the enhanced image based on the live image data or the one or more modification parameters in order to obtain the enhanced image. [A5] The method of Embodiment A4, wherein the one or more modification parameters include one or more conversion parameters, brightness parameters, contrast parameters, saturation parameters, or sharpness parameters. [A6] A method of any embodiment A1 to A5, wherein the step of obtaining feedback on the set of stimuli includes the step of obtaining an image of the eye captured during the presentation, the image of the eye being an image of the user's eye, and the step of providing feedback on the set of stimuli includes the step of providing the image of the eye to the model. [A7] The method of Embodiment A5, wherein the image of the eye is an image of the eyeball, an image of the retina of the eye, or an image of the cornea of ​​the eye. [A8] A method according to any embodiment A1 to A7, wherein the step of obtaining feedback on the set of stimuli includes the step of obtaining a display of the user's response to one or more stimuli from the set of stimuli, or a display of the user's lack of response to one or more stimuli from the set of stimuli, and the step of providing feedback on the set of stimuli includes the step of providing the display of the response or the display of the lack of response to the model. [A9] The method of Embodiment A8, wherein the response includes eye movements, gaze direction, changes in pupil size, or user modifications to one or more stimuli by user input from the user. [A10] The method of Embodiment A9, wherein the user modification includes the movement of one or more stimuli by the user's user input, or supplemental data provided by the user's user input for one or more stimuli displayed to the user. [A11] A method of any embodiment A1 to A10, further comprising: a step of obtaining a second set of stimuli, the second set of stimuli being generated based on the model's processing of the set of stimuli and feedback relating to the set of stimuli; a step of displaying the second set of stimuli to the user; a step of obtaining feedback relating to the second set of stimuli (for example, the feedback indicating whether or how the user is viewing one or more stimuli from the second set); and a step of providing the feedback relating to the second set of stimuli to the model, the model being further configured based on the feedback relating to the second set of stimuli. [A12] A method of any embodiment A1 to A11, further comprising the step of determining a defective visual field portion of the user's visual field by the model based on feedback relating to the set of stimuli, wherein the user's visual field comprises a plurality of visual field portions, and the defective visual field portion is one of the plurality of visual field portions that does not satisfy one or more visual criteria. [A13] The method of Embodiment A12, wherein the enhanced image is based on one or more transformations corresponding to the defective field of view portion of the live image data, and the image portion of the live image data is represented in the image portion of the enhanced image that is outside the defective field of view portion. [A14] The enhanced image is based on one or more modifications to the brightness or contrast of the live image data, wherein (i) the increase in brightness, contrast, or sharpness level is applied to the image portion of the live image data corresponding to the defective field of view portion to generate the corresponding image portion of the enhanced image, and (ii) the increase in brightness, contrast, or sharpness level is not applied to other image portions of the live image data to generate the corresponding image portion of the enhanced image, according to any of the methods of Embodiments A12 to A13. [A15] A method of any embodiment A12 to A14, further comprising: detecting an object (for example, located in or expected to be located in the defective field of view); determining that the object is not sufficiently depicted in any image portion of the enhanced image corresponding to at least one of the field of view portions that satisfy the one or more visual criteria; generating a prediction that the object will physically come into contact with the user; and displaying a warning (for example, on the enhanced image) based on the prediction of physical contact and the determination that the object is not sufficiently depicted in any image portion of the enhanced image corresponding to at least one of the field of view portions that satisfy the one or more visual criteria, wherein the warning indicates the direction in which the object is approaching. [A16] Any method of Embodiments A1 to A15, wherein one or more of the steps described above are performed by a wearable device. [A17] The method of Embodiment A16, wherein the wearable device includes one or more cameras configured to capture the live image data and one or more display portions configured to display one or more enhanced images. [A18] The method according to any one of embodiments A16 to A17, wherein the one or more display portions include first and second display portions of the wearable device. [A19] The method of Embodiment A18, wherein the wearable device includes a first monitor including the first display portion and a second monitor including the second display portion. [A20] The method of any embodiment A16 to A19, wherein the one or more display portions include one or more dynamic display portions on one or more transparent displays of the wearable device, and one or more enhanced images are displayed on the one or more display portions. [A21] Any method of embodiment A1 to A20, further comprising the step of monitoring one or more changes relating to one or more of the user's eyes. [A22] The method of Embodiment A21, further comprising the steps of: providing the one or more changes to the model as further feedback; obtaining one or more modification parameters from the model based on the live image data, eye characteristic information, or environmental characteristic information provided to the model; and generating the enhanced image based on the live image data and the one or more modification parameters in order to obtain the enhanced image. [A23] Any method of Embodiments A21 to A22, further comprising the step of adjusting the position, shape, size, or transparency of one or more of the first or second display portions on one or more transparent displays of the wearable device based on the monitoring, wherein the step of displaying the enhanced image includes the step of displaying the enhanced image on the first or second display portion. [A24] The model includes a neural network or other machine learning model, according to any of the methods of Embodiments A1 to A23. [B1] A method comprising the steps of: acquiring multiple images of a scene; determining a region common to the multiple images; determining a region of each of the multiple images that has a difference from a corresponding region of at least one other image among the multiple images; generating an enhanced image based on the common region and the region having the difference; and displaying the enhanced image. [B2] The method of Embodiment B1, wherein the step of generating the enhanced image includes the step of generating the enhanced image based on the common region and the difference region, wherein (i) the first region of the enhanced image includes a representation of the common region, (ii) the second region of the enhanced image includes a representation of the difference region, and (iii) the second region surrounds the first region in the enhanced image. [B3] The method of Embodiment B2, wherein the step of generating the enhanced image includes the step of generating the enhanced image based on the common region, the difference region, and a second region common to the plurality of images, wherein (i) the first region of the enhanced image includes a representation of the common region and a representation of the second common region, and (ii) the second region of the enhanced image includes a representation of the difference region. [B4] The method according to any one of Embodiments B1 to B3, wherein the common region is a region of at least one image among the plurality of images that corresponds to the macular region of the visual field of the eye, or a region within the macular region of the visual field. [B5] The method according to any one of embodiments B1 to B4, wherein each of the difference regions is a region of at least one image among the plurality of images that corresponds to a peripheral region of the field of vision of the eye or a region within the peripheral region of the field of vision. [B6] A method according to any of embodiments B1 to B5, further comprising the step of shifting each of the plurality of images, wherein the step of generating the enhanced image includes, following the step of performing the shift, the step of generating the enhanced image based on the common region and the difference region. [B7] The method of Embodiment B6, wherein the step of performing the shift includes the step of shifting each of the plurality of images such that the size of the common area decreases and the size of at least one of the difference areas increases. [B8] A method of any embodiment B1 to B7, further comprising the step of resizing one or more regions of the plurality of images, wherein the step of generating the enhanced image includes, following the resizing step, the step of generating the enhanced image based on the common region and the difference region. [B9] The method of Embodiment B8, wherein the step of resizing includes resizing one or more regions of the plurality of images such that the range of any resizing of the common region is different from the range of any resizing of at least one of the difference regions. [B10] The method according to any one of embodiments B8 to B9, wherein the resizing step includes resizing one or more regions of the plurality of images such that the rate of change of the size of the common region represented in the first region of the enhanced image is greater than or less than the rate of change of the size of at least one of the difference regions represented in the second region of the enhanced image. [B11] The method of embodiment B10, wherein the rate of change of the size of at least one of the difference regions is 0, and the rate of change of the size of the common region is greater than 0. [B12] The method of Embodiment B10, wherein the rate of change of the size of at least one of the difference regions is greater than 0, and the rate of change of the size of the common region is 0. [B13] A method of any embodiment B1 to B12, further comprising the step of performing a fisheye transformation, an conformal transformation, or other transformation on the common region, wherein the step of generating the enhanced image includes, following the step of performing the transformation, the step of generating the enhanced image based on the common region and the difference region. [B14] Any method of Embodiments B1 to B13, further comprising the step of determining a defective field of view portion of the user's field of view, wherein the user's field of view includes a field of view portion, the defective field of view portion is one of the field of view portions that does not meet one or more visual criteria, the enhanced image is generated based on the determined defective field of view portion, and at least one of the common region or the difference region in the enhanced image does not overlap with the defective field of view portion of the user's field of view. [B15] A method in any of embodiments B1 to B14, further comprising the step of determining a portion of the user's field of view that satisfies (i) one or more visual criteria, (ii) one or more positional criteria, and (iii) one or more size criteria, wherein an enhanced image is generated based on the portion of the field of view, and at least one of the common region or the difference region in the enhanced image is located within the portion of the field of view. [B16] The method of Embodiment B15, wherein the one or more size criteria include the requirement that the field of view portion is the largest field of view portion of the user's field of view that satisfies the one or more visual criteria and the one or more positional criteria. [B17] Any method of Embodiments B15 to B16, wherein the one or more positional references include the requirement that the center of the field of view corresponds to a point in the macular region of the user's eye. [B18] Any method of Embodiments B1 to B17, wherein one or more of the steps described above are performed by a wearable device. [B19] The method of Embodiment B18, further comprising the step of making one or more display portions of the wearable device transparent, wherein the step of displaying the enhanced image includes the step of displaying the enhanced image on one or more other display portions of the wearable device other than the one or more transparent display portions. [B20] The method of embodiment B19, further comprising the step of adjusting the one or more transparent display portions and the one or more other display portions of the wearable device. [B21] The method of Embodiment B20, further comprising the step of monitoring one or more changes relating to one or more of the user's eyes, wherein the adjustment step includes adjusting the one or more transparent display portions and the one or more other display portions of the wearable device based on the monitoring. [B21] The method of Embodiment B20, further comprising the step of monitoring one or more changes relating to one or more of the user's eyes, wherein the adjustment step includes adjusting the one or more transparent display portions and the one or more other display portions of the wearable device based on the monitoring. [B22] The method according to any one of B20 to B21, wherein the adjustment step includes adjusting the position, shape, size, or transparency of one or more transparent display portions of the wearable device based on the monitoring. [B23] The enhanced image or adjustment is based on one or more of the changes, by any method of Embodiments B20 to B22. [B24] The step of displaying the enhanced image is to display one or both of the common area or the difference area on one or more of the other display parts of the wearable device, and at least one of the common area or the difference area is not displayed on one or more of the transparent display parts of the wearable device, according to any of embodiments B18 to B23. [B25] The wearable device includes first and second cameras, and the step of obtaining the plurality of images includes the step of obtaining at least one of the plurality of images with the first camera of the wearable device, and the step of obtaining at least one other image of the plurality of images with the second camera of the wearable device, according to any one of embodiments B18 to B24. [B26] The method according to any of embodiments B18 to B25, wherein one or more monitors of the wearable device include first and second monitors, and the step of displaying the enhanced image includes the step of displaying the enhanced image using the first and second monitors. [B27] The wearable device includes a wearable eyeglasses device, according to any of the embodiments B18 to B26. [B28] The enhanced image or adjustment is based on any of the methods of Embodiments B1 to B27, wherein the enhancement image or adjustment is based on feedback regarding a set of stimuli (e.g., feedback indicating whether or how the user is perceiving one or more stimuli). [C1] A method comprising the steps of monitoring one or more changes relating to one or more of a user's eyes; adjusting one or more transparent display portions or one or more other display portions of a wearable device based on the monitoring; and displaying an enhanced image on one or more other display portions of the wearable device, wherein the enhanced image is based on live image data acquired via the wearable device. [C2] The method of the embodiment C1, wherein the adjustment method includes the step of adjusting the position, shape, size, brightness level, contrast level, sharpness level, or saturation level of one or more transparent display portions of the wearable device, or one or more other display portions of the wearable device, based on the monitoring. [C3] Any method of Embodiments C1 to C2, further comprising the step of determining a defective field of view portion of the user's field of view, wherein the user's field of view includes a field of view portion, the defective field of view portion is one of the field of view portions that does not meet one or more visual criteria, and the adjustment step includes adjusting one or more transparent display portions of the wearable device to the position, shape, or size of one or more transparent display portions such that one or more transparent display portions of the wearable device do not overlap with the defective field of view portion. [C4] The method of Embodiment C3, further comprising: detecting an object (for example, located in or expected to be located in the defective field of view); determining that the object is not sufficiently depicted in any image portion of the enhanced image corresponding to at least one of the field of view portions that satisfy one or more visual criteria; generating a prediction that the object will physically come into contact with the user; and displaying a warning (for example, on the enhanced image) based on the prediction of physical contact and the determination that the object is not sufficiently depicted in any image portion of the enhanced image corresponding to at least one of the field of view portions that satisfy the one or more visual criteria, wherein the warning indicates the direction in which the object is approaching. [C5] A method of any embodiment C1 to C4, further comprising the steps of: providing a model with information relating to one or more eyes, the model being constructed based on the information relating to one or more eyes; and, following the construction of the model, providing the model with one or more monitored changes relating to the one or more eyes in order to obtain a set of modification parameters, wherein the step of adjusting the one or more transparent display portions includes adjusting the one or more transparent display portions based on one or more modification parameters from the set of modification parameters. [C6] The method of Embodiment C5, wherein the information relating to one or more eyes includes one or more images of the one or more eyes. [C7] The information relating to one or more eyes is provided by any of the methods of Embodiments C5 to C6, including feedback relating to a set of stimuli (e.g., feedback indicating whether or how the user is perceiving one or more stimuli). [C8] The method of any of Embodiments C1 to C7, wherein the one or more changes include eye movements, changes in gaze direction, or changes in pupil size. [C9] The enhanced image or adjustment is based on any of the methods of Embodiments C1 to C8, wherein the enhancement image or adjustment is based on feedback regarding a set of stimuli (e.g., feedback indicating whether or how the user is perceiving one or more stimuli). [C10] The enhanced image or adjustment is based on one or more of the changes, by any method of Embodiments C10 to C9. [C11] The adjustment is performed simultaneously with the display of the enhanced image, in any of the methods of Embodiments C1 to C10. [C12] A method according to any of embodiments C1 to C11, wherein one or more of the steps described above are performed by the wearable device. [C13] The wearable device includes a wearable eyeglasses device, according to any of the embodiments C1 to C12. [D1] A method comprising: monitoring one or more of a user's eyes (for example, during a first monitoring period in which a set of stimuli is displayed to the user); obtaining feedback on the set of stimuli (for example, during the first monitoring period); and generating a set of correction profiles associated with the user based on the feedback on the set of stimuli, wherein each correction profile of the set of correction profiles includes (i) a set of eye-related characteristics and (ii) one or more correction parameters applied to an image to correct an image for the user when the eye-related characteristics of the user match the set of associated eye-related characteristics. [D2] The method of Embodiment D1, wherein the feedback relating to the set of stimuli indicates that the user is looking at or how they are looking at one or more stimuli from the set of stimuli. [D3] The feedback relating to the set of stimuli includes one or more characteristics relating to one or more eyes that occur when the one or more stimuli are displayed (for example, during the first monitoring period), according to any one of Embodiments D1 to D2. [D4] Any method of Embodiments D1 to D3, further comprising: monitoring one or more of the user's eyes (for example, during a second monitoring period); acquiring image data representing the user's environment (for example, during the second monitoring period); acquiring one or more modification profiles associated with the user (for example, from the second monitoring period) based on (i) the image data or (ii) the characteristics relating to the one or more of the eyes; and displaying modified image data to the user (for example, during the second monitoring period) based on (i) the image data and (ii) the one or more modification profiles. [D5] The method of Embodiment D4, wherein the characteristics relating to one or more eyes include the direction of gaze, pupil size, marginal position, visual axis, optical axis, or position or movement of the eyelid. [D6] Any method of Embodiments D1 to D5, wherein the step of obtaining feedback on the set of stimuli includes the step of obtaining an image of the eye captured during the first monitoring period, the image of the eye being an image of the user's eye, and the step of generating the set of correction profiles includes the step of generating the set of correction profiles based on the image of the eye. [D7] The method of embodiment D6, wherein the image of the eye is an image of the retina of the eye or an image of the cornea of ​​the eye. [D8] A method according to any embodiment D1 to D7, wherein the step of obtaining feedback on the set of stimuli includes the step of obtaining an indication of the user's response to one or more stimuli or an indication of the user's lack of response to one or more stimuli, and the step of generating a set of correction profiles includes the step of generating a set of correction profiles based on the indication of the response or the indication of the lack of response. [D9] The method of Embodiment D8, wherein the response includes eye movement, a change in gaze direction, or a change in pupil size. [D10] A method according to any of embodiments D1 to D9, wherein one or more of the steps described above are performed by a wearable device. [D11] The method of embodiment D10, wherein the wearable device includes a wearable eyeglasses device. [E1] A method comprising the steps of: displaying a first stimulus at a first interface position on a user interface of a user based on a fixation point for visual examination presentation; adjusting the fixation point for visual examination presentation during the visual examination presentation based on eye characteristic information relating to the user, wherein the eye characteristic information indicates one or more characteristics relating to one or more of the user's eyes that occurred during the visual examination presentation; displaying a second stimulus at a second interface position on the user interface based on the adjusted fixation point for visual examination presentation; acquiring feedback information indicating feedback regarding the first stimulus and feedback regarding the second stimulus, wherein the feedback regarding the first or second stimulus indicates the user's response to the first or second stimulus or the lack of a response by the user; and generating visual defect information associated with the user based on the feedback information. [E2] The method of Embodiment E1, wherein the user interface is configured to display a view having a horizontal dimension corresponding to a first degree or a vertical dimension corresponding to the first degree, and the visual defect information has a coverage area greater than the first degree with respect to the horizontal dimension of the user's field of view or with respect to the vertical dimension of the user's field of view. [E3] The user interface is configured to display a view having a given dimension corresponding to a first degree, and the visual defect information is generated such that (i) the visual defect information indicates at least two defects present at a field of view position in the user's field of view, and (ii) the field of view position is separated by more than the first degree with respect to the given dimension of the user's field of view, according to any one of embodiments E1 to E2. [E4] The user interface is configured to display a view having a given dimension corresponding to a first degree, the feedback information further indicates feedback regarding a third stimulus displayed on the user interface during the visual examination presentation, and the method comprises the steps of determining, based on the feedback information, whether a visual defect exists at a visual field position in the user's visual field such that at least two of the visual field positions are separated from each other by more than the first degree with respect to the given dimension of the visual field, and generating the visual defect information based on the determination of whether a visual defect exists at the visual field position, one of the methods of any embodiment E1 to E3. [E5] A method according to any one embodiment E1 to E4, further comprising the steps of: determining a first interface position of a first stimulus based on a fixation point for the visual examination presentation and a first relative position associated with the first stimulus; and determining a second interface position of a second stimulus based on a adjusted fixation point for the visual examination presentation and a second relative position associated with the second stimulus, wherein the step of displaying the first stimulus includes, during the visual examination presentation, displaying the first stimulus at the first interface position on the user interface based on the determination of the first interface position; and the step of displaying the second stimulus includes, during the visual examination presentation, displaying the second stimulus at the second interface position on the user interface based on the determination of the second interface position. [E6] A method of any embodiment E1 to E5, further comprising the step of selecting the first interface position for a first stimulus during the presentation of the visual test, on the basis that the first interface position is further from the fixation point than one or more other interface positions on the user interface, wherein the one or more other interface positions correspond to one or more other visual field positions in the test set, and the step of displaying the first stimulus includes, during the presentation of the visual test, displaying the first stimulus at the first interface position on the user interface based on the selection of the first interface position. [E7] The method of embodiment E6, further comprising the step of removing the first field of view position from the inspection set. [E8] The method of Embodiment E7, wherein the step of deleting the first visual field position includes deleting the first visual field position from the test set such that the first visual field position cannot be selected from the test set during the presentation of the visual test. [E9] A method of any embodiment E7 to E8, further comprising the step of removing the first visual field position from the test set, the step of selecting the second interface position for a second stimulus on the basis that the second interface position is further from the adjusted fixation point than one or more other interface positions on the user interface, and the step of displaying the second stimulus, the step of displaying the second stimulus at the second interface position on the user interface on the basis of the selection of the second interface position during the visual test presentation. [E10] A method of any embodiment E6 to E9, wherein the step of selecting the first interface position includes selecting the first interface position for the first stimulus on the basis that the first interface position is located at least the same distance from the fixation point as all other interface positions on the user interface corresponding to the field of view positions of the test set other than the first field of view position, with respect to a given dimension. [E11] A method of any embodiment E6 to E10, wherein the step of selecting the second interface position includes selecting the second interface position for the second stimulus on the basis that the second interface position is located at least as far away from the adjusted fixation point as all other interface positions on the user interface corresponding to the field of view positions of the test set other than the second field of view position, with respect to a given dimension. [E12] A method of any embodiment E1 to E11, further comprising: locking the adjusted fixation point, wherein readjustment of the fixation point is avoided while the adjusted fixation point is locked; displaying one or more stimuli on the user interface based on the adjusted fixation point while the adjusted fixation point is locked; and unlocking the adjusted fixation point before displaying the second stimuli. [E13] Any method of Embodiments E1 to E12, further comprising the step of displaying a plurality of stimuli on the user interface while the adjusted fixation point is in the same position (e.g., the first interface position), and then de-emphasizing or removing the plurality of stimuli from the user interface, wherein at least one of the plurality of stimuli is displayed on the user interface following the display of at least one other of the plurality of stimuli on the user interface. [E14] The method of Embodiment E13, wherein the plurality of stimuli are de-emphasized or removed after being displayed, while the first stimuli remains displayed at the first interface position on the user interface. [E15] Any method of embodiments E13 to E14, further comprising the steps of de-emphasizing or removing the first stimulus from the user interface, following the display of at least one of the plurality of stimuli on the user interface, and then highlighting or re-displaying the first stimulus at the first interface position on the user interface. [E16] The eye characteristic information is a method according to any of Embodiments E1 to E15, which indicates one or more gaze directions of the user, changes in pupil size, eyelid movements, head movements, or other eye-related characteristics that occurred during the presentation of the visual examination. [F1] A method comprising the steps of: monitoring characteristics of a user's eye during a visual examination presentation via two or more user interfaces provided to each eye (e.g., on two or more displays), wherein the user's eye includes a first eye and a second eye; presenting one or more stimuli at one or more locations on at least one of the user interfaces; and determining visual defect information of the first eye based on characteristics of one or more eyes (e.g., the first eye) that occurred during the presentation of the stimuli. [F2] The method of Embodiment F1, wherein the step of determining the visual defect information includes the step of determining the deviation measurement of the first eye based on the characteristics of one or more of the first eyes that occurred during the presentation of the stimulus. [F3] The method of embodiment F2, wherein the deviation measurement value indicates the deviation of the first eye relative to the second eye. [F4] The step of presenting the stimulus includes the step of presenting the stimulus at a position on a first user interface used for the first eye at a first time, wherein the presentation of the stimulus occurs while no stimulus is being presented on a second user interface used for the second eye, according to any of embodiments F1 to F3. [F5] The step of presenting the stimulus includes the step of presenting the stimulus at a position on the first user interface while the stimulus intensity of the second user interface does not meet the stimulus intensity threshold, according to any one of embodiments F1 to F4. [F6] Any method of embodiment F4 to F5, further comprising the step of presenting a stimulus at the position on the second user interface at a prior time (before the first time) while the stimulus is not presented on the first user interface. [F7] A method according to any one of embodiments F4 to F6, further comprising the steps of: presenting a stimulus at a first position on the first display and presenting a stimulus at a first position on the second display during a prior time before the first time; detecting that the first eye does not fixate on the first position when the stimulus is presented on the first display during the prior time; and determining that the user's first eye is a deviated eye based on the detection that the first eye is not fixating. [F8] A method according to any embodiment F4 to F7, further comprising the steps of: causing a stimulus to be presented at a modified position on the first display during a subsequent time following the first time, based on the visual defect information (e.g., the deviation measurement), wherein the presentation during the subsequent time occurs while the stimulus is not presented on the second display, and the modified position is different from the first position; and determining the visual defect information (e.g., the deviation measurement) based on the characteristics of one or more of the first or second eyes that do not change beyond a change threshold during the presentation during the subsequent time. [F9] The method of embodiment F8, further comprising the step of determining the corrected position as the position in which a stimulus is presented on the first display in the subsequent time, based on the visual defect information (e.g., the deviation measurement). [F10] The method according to any one of embodiments F1 to F2, wherein the step of presenting the stimulus includes the step of presenting the stimulus at a position on a first user interface used for the first eye and at a corresponding position on a second user interface used for the second eye at a given time. [F11] A method of any embodiment F1 to F10, further comprising the step of generating a correction profile associated with the user based on the visual defect information (e.g., the deviation measurement), wherein the correction profile includes one or more correction parameters applied to correct an image for the user. [F12] The method of embodiment F11, further comprising the steps of (i) displaying modified video stream data to the user based on a modified profile associated with the user, and (ii) displaying modified video stream data to the user. [F13] The method of Embodiment F12, wherein the modification profile includes translational or rotational parameters applied to modify an image for the first eye when the line of sight of the second eye is directed toward the first position, and the step of displaying the modified video stream data includes detecting that the line of sight of the second eye is directed toward the first position, modifying the video stream data based on the detection of the line of sight of the second eye using the translational or rotational parameters to generate the modified video stream data, and displaying the modified video stream data to the user's first eye. [F14] A method according to any one of embodiments F1 to F13, further comprising: generating a first correction profile associated with the user based on the deviation measurement, the first correction profile including one or more correction parameters applied to correct an image for the first eye in response that the line of sight direction of the second eye is directed to the first position; and generating a second correction profile based on a second deviation measurement of the first eye, the second correction profile including one or more correction parameters applied to correct an image for the first eye in response that the line of sight direction of the second eye is directed to a second position different from the first position. [F15] The method of any embodiment F1 to F14, wherein the step of determining the visual defect information includes the step of determining whether the user has diplopia or the degree of diplopia based on the number or type of stimuli the user sees. [F16] The method of embodiment F15, further comprising the step of determining the number or type of stimuli the user sees based on user input indicating the number or type of stimuli the user sees. [F17] Any method of embodiment F15 to F16, further comprising the step of determining the number or type of stimuli the user sees based on the characteristics of one or more eyes that occur when the stimuli are presented. [F18] The step of determining the visual defect information includes any of the methods of embodiments F1 to F17, which includes determining whether the user has stereopsis or the degree of stereopsis based on the characteristics of one or more eyes that occur when the stimulus is presented. [F19] The eye-related characteristics include one or more gaze directions, changes in pupil size, or other eye-related characteristics, as described in any of the embodiments F1 to F18. [G1] A tangible, non-temporary, machine-readable medium storing an instruction, wherein when the instruction is executed by a data processing device, the data processing device performs a process including any of the steps of Embodiments A1 to A24, B1 to B28, C1 to C13, D1 to D11, E1 to E16, or F1 to F19. [G2] A system comprising one or more processors and a memory storing instructions, wherein when an instruction is executed by the one or more processors, the one or more processors realize a process including any of the steps in Embodiments A1 to A24, B1 to B28, C1 to C13, D1 to D11, E1 to E16, or F1 to F19.

Claims

1. A system for facilitating the generation of visual defect information using dynamic fixation points, A computer system having one or more processors programmed using computer program instructions, wherein when the computer program instructions are executed, the computer system The process involves displaying a first stimulus at a first interface position on the user interface of the user's wearable device during the presentation of a visual test, During the presentation of the visual examination, based on the fact that the eye characteristic information related to the user indicates that the user is looking at the first interface position, the fixation point for the presentation of the visual examination is adjusted to the first interface position. A step of locking the fixation point at the first interface position, wherein readjustment of the fixation point is avoided while the fixation point is locked, During the presentation of the visual test, a step of displaying one or more stimuli at one or more interface positions on the user interface other than the first interface position and the second interface position, based on the fixation point at the first interface position, wherein the one or more stimuli are displayed at the one or more interface positions on the user interface while the fixation point is locked. A step of unlocking the aforementioned fixation point, The process involves, during the presentation of the visual test, after the lock on the fixation point is released, displaying a second stimulus at the second interface position on the user interface; During the presentation of the visual examination, the fixation point is adjusted to the second interface position based on the fact that the characteristic information of the eye indicates that the user is looking at the second interface position. During the presentation of the visual test, a step is to acquire feedback information indicating feedback regarding the first stimulus, feedback regarding one or more stimuli, and feedback regarding the second stimulus. A step of generating visual impairment information associated with the user based on the aforementioned feedback information, A system that executes this.

2. The step of displaying one or more stimuli is, The process includes, while the fixation point is at the first interface position, displaying a plurality of stimuli at an interface position different from the first interface position, and then reducing the intensity of the display of the plurality of stimuli on the user interface or removing them from the user interface, At least one of the plurality of stimuli is displayed on the user interface following the display of at least one other stimulus from the plurality of stimuli on the user interface. The system according to claim 1.

3. The system according to claim 2, wherein after the plurality of stimuli are displayed, the intensity of the display is reduced or removed while the first stimulus continues to be displayed at the first interface position on the user interface.

4. The system according to claim 2, wherein the computer system, following the display of at least one of the plurality of stimuli on the user interface, further performs the steps of reducing the intensity of the display of the first stimulus on the user interface or removing it from the user interface, and then increasing the intensity of the display of the first stimulus at the first interface location on the user interface or redisplaying it.

5. The system according to claim 1, wherein the first interface position is selected for the first stimulus on the basis that the first interface position is one of the positions on the user interface furthest from the fixation point before the first stimulus is displayed at the first interface position.

6. The system according to claim 5, wherein the second interface position is selected for the second stimulus on the basis that the second interface position is one of the positions on the user interface furthest from the fixation point adjusted to the first interface position.

7. The system according to claim 1, wherein the eye characteristic information indicates one or more of the user's gaze directions, changes in pupil size, eyelid movements, or head movements that occurred during the presentation of the visual examination.

8. A method implemented by one or more processors that execute computer program instructions, wherein when the computer program instructions are executed, the method is executed, The process involves displaying a first stimulus at a first interface position on the user interface of the user's wearable device during the presentation of a visual test, During the presentation of the visual examination, based on the fact that the eye characteristic information related to the user indicates that the user is looking at the first interface position, the fixation point for the presentation of the visual examination is adjusted to the first interface position. A step of locking the fixation point at the first interface position, wherein readjustment of the fixation point is avoided while the fixation point is locked, During the presentation of the visual test, a step of displaying one or more stimuli at one or more interface positions on the user interface other than the first interface position and the second interface position, based on the fixation point at the first interface position, wherein the one or more stimuli are displayed at the one or more interface positions on the user interface while the fixation point is locked. A step of unlocking the aforementioned fixation point, During the presentation of the visual test, after the lock on the fixation point is released, a second stimulus is displayed at the second interface position on the user interface. During the presentation of the visual examination, the fixation point is adjusted to the second interface position based on the fact that the characteristic information of the eye indicates that the user is looking at the second interface position. During the presentation of the visual test, a step is taken to acquire feedback information indicating feedback regarding the first stimulus, feedback regarding one or more stimuli, and feedback regarding the second stimulus. A step of generating visual impairment information associated with the user based on the aforementioned feedback information, A method that includes [a certain feature].

9. The step of displaying one or more stimuli is, The process includes, while the fixation point is at the first interface position, displaying a plurality of stimuli at an interface position different from the first interface position, and then reducing the intensity of the display of the plurality of stimuli on the user interface or removing them from the user interface, At least one of the plurality of stimuli is displayed on the user interface following the display of at least one other stimulus from the plurality of stimuli on the user interface. The method according to claim 8.

10. The method according to claim 9, wherein, after the plurality of stimuli are displayed, the intensity of the display is reduced or removed while the first stimulus continues to be displayed at the first interface position on the user interface.

11. The method according to claim 9, further comprising the steps of: displaying at least one of the plurality of stimuli on the user interface; then reducing the intensity of the display of the first stimulus on the user interface or removing it from the user interface; and then increasing the intensity of the display of the first stimulus at the first interface location on the user interface or redisplaying it.

12. The method according to claim 8, wherein the first interface position is selected for the first stimulus on the basis that the first interface position is one of the positions on the user interface furthest from the fixation point before the first stimulus is displayed at the first interface position.

13. The method according to claim 12, wherein the second interface position is selected for the second stimulus based on the fact that the second interface position is one of the positions on the user interface furthest from the fixation point adjusted to the first interface position.

14. The method according to claim 8, wherein the eye characteristic information indicates one or more of the user's gaze directions, changes in pupil size, eyelid movements, or head movements that occurred during the presentation of the visual examination.

15. One or more non-temporary computer-readable storage media comprising instructions, wherein when the instructions are executed by one or more processors, The process involves displaying a first stimulus at a first interface position on the user interface of the user's wearable device during the presentation of a visual test, During the presentation of the visual examination, based on the fact that the eye characteristic information related to the user indicates that the user is looking at the first interface position, the fixation point for the presentation of the visual examination is adjusted to the first interface position. A step of locking the fixation point at the first interface position, wherein readjustment of the fixation point is avoided while the fixation point is locked, During the presentation of the visual test, a step of displaying one or more stimuli at one or more interface positions on the user interface other than the first interface position and the second interface position, based on the fixation point at the first interface position, wherein the one or more stimuli are displayed at the one or more interface positions on the user interface while the fixation point is locked. A step of unlocking the aforementioned fixation point, During the presentation of the visual test, after the lock on the fixation point is released, a second stimulus is displayed at the second interface position on the user interface. During the presentation of the visual examination, the fixation point is adjusted to the second interface position based on the fact that the characteristic information of the eye indicates that the user is looking at the second interface position. During the presentation of the visual test, a step is taken to acquire feedback information indicating feedback regarding the first stimulus, feedback regarding one or more stimuli, and feedback regarding the second stimulus. A step of generating visual defect information associated with the user based on the aforementioned feedback information, One or more non-temporary computer-readable storage media on which processing including the above is performed.

Images