Methods to screen and quantify thyroid eye disease in a virtual reality headset

Abstract

One aspect of the disclosure here is a virtual reality headset-based system that performs a vision test or eye examination (exam) in an automated and repeatable manner that quantifies the tested user's degree of, or likelihood of suffering from, a stage of Thyroid Eye Disease, TED. The system may be configured to analyze images of the user's eyes to determine various quantifiable metrics, where these metrics can be used to, for example, determine how the management of the disease is progressing. Other aspects are also described.

Description

METHODS TO SCREEN AND QUANTIFY THYROID EYE DISEASE IN A VIRTUAL REALITY HEADSETCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 573,455, filed on April 2, 2024, the contents of which are incorporated herein by reference.FIELD

[0002] An aspect of the disclosure here relates to portable head worn equipment that can be used for performing vision testing of the wearer’s eyes, and in particular for Thyroid Eye Disease, TED, screening.BACKGROUND

[0003] TED screening is a very intensive, manual process and involves myriad different measurements. These include physical exams (to measure proptosis, strabismus, etc.), blood sample analysis for testing hormone levels, and imaging (ultrasound or computed tomography scan, to measure anatomy of eye structures). These methods are time-consuming, expensive, and rely on clinicians to make accurate measurements. As a result, these steps are typically taken when the disease has already reached a certain degree of severity. Screening and quantifying potential TED patients early in their disease progression would lead to lower complications and better outcomes.SUMMARY

[0004] One aspect of the disclosure here is a virtual reality headset-based system that performs a vision test or examination (exam) in an automated and repeatable manner that quantifies the tested user’s degree of, or likelihood of suffering from, a stage of TED (also referred to here as a classification). The system may also be configured to analyze images of the user’s eyes to determine various quantifiable metrics, where these metrics can be used to, for example determine how the management of the disease is progressing.

[0005] In one aspect, a series of measurements or captures of a user’s eyes are made by an internal camera of a virtual reality, VR, headset, as the user wearing the VR headset is presented with stimuli appearing on a display inside the VR headset at which they are looking. The VR headset may improve consistency and ease of conducting the test in various ambient light environments, in a more efficient (less time consuming) manner. The stimuli are presented in a light-controlled environment of the VR headset, which can be regulated. The presentation of the stimuli is flexible, including the ability to present different stimuli to each eye. Also, the capture of the eye behavior may be by imaging in the near-infrared, NIR, which works even when the user is not seeing anything. The optics for viewing the stimuli that may be optimized for a desired field of view and capture resolution. There is also the advantage of device consistency because the system has knowledge of how far away the display of the VR headset is from the user’s eyes and thus the results of tests performed at different sittings will be consistent (as opposed to measurements taken by a clinician, which will have some variability).

[0006] The results of the test may be used by, for example, an eye care professional to diagnose TED in the user that might call for additional testing or a recommended treatment. The system may also be configured to analyze images of the user's eyes to determine various quantifiable metrics, where these metrics can be used to determine how the management of the disease is progressing over time.

[0007] The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the disclosure includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those disclosed in the Detailed Description below7and particularly pointed out in the Claims section. Such combinations may have advantages that are not recited in the above summary.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Several aspects of the disclosure here are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” aspect in this disclosure are not necessarily to the same aspect, and they mean at leastone. Also, in the interest of conciseness and reducing the total number of figures, a given figure may be used to illustrate the features of more than one aspect of the disclosure, and not all elements in the figure may be required for a given aspect.

[0009] Fig. 1 is a diagram of an example virtual reality, VR, headset-based system for TED vision testing.

[0010] Fig. 2 is a flow diagram of an example method for TED vision testing.

[0011] Fig. 3 illustrates an example of several internal cameras in the headset used for TED vision testing.

[0012] Fig. 4 shows an example of proptosis measurement using a side view internal camera.

[0013] Fig. 5 illustrates an example of several metrics obtained from eye image segmentation that are useful for TED diagnosis.

[0014] Fig. 6 illustrates how a smartphone app and the headset work together to provide a more continuous monitoring of TED.DETAILED DESCRIPTION

[0015] Several aspects of the disclosure with reference to the appended drawings are now explained. Whenever the shapes, relative positions and other aspects of the parts described are not explicitly defined, the scope of the invention is not limited only to the parts shown, which are meant merely for the purpose of illustration. Also, while numerous details are set forth, it is understood that some aspects of the disclosure may be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

[0016] Fig. 1 is a diagram of an example virtual reality, VR, headset-based system that can be used for TED testing. The system is an example of a stereoscopic system, where some of the aspects described below are also applicable in other stereoscopic systems such those that use a lenticular array. The system in Fig. 1 is composed of a VR headset 1 which 1 is fitted over the eyes of a user (person or observer) as shown. Ithas a wired or wireless communication network interface for communicating data with an external computing device 9, e.g., a tablet computer, a laptop computer, etc. A human operator, such as an eye care professional, ECP, may interact briefly with software that is being executed by one or more microelectronic data processors (generically, “a processor’') of the system to conduct the vision test. Once launched or initialized the software may conduct the test automatically (without input from the operator) by controlling the various electronic and optical components of the VR headset 1. The software may have components that are executed by a processor that is in the VR headset 1, and it may have components that are executed by a processor which is part of the external computing device 9. Some of these software components may be executed either in the VR headset 1 or in the external computing device 9. The software may interact with the operator through a graphical user interface component that uses for example a touchscreen of the external computing device 9 for presenting results of the vision test and receiving information from the operator.

[0017] The VR headset 1 may have a form factor like goggles, as shown, that blocks all ambient lighting outside of the VR headset 1 so as to create a light controlled environment around the user’s eyes (that is independent of the ambient lighting outside of the VR headset 1). The VR headset 1 may be composed of a left visible light display 3 to which a left compartment 5 is coupled that fits over the left eye of the user, and a right visible light display 4 to which a right compartment 6 is coupled that fits over the right eye of the user. The left and right compartments are configured, e.g., shaped and being opaque, so that the user cannot see the right display 4 using only their left eye, and the user cannot see the left visible light display 3 using only their right eye (once the VR headset 1 has been fitted over the user’s eyes). Also, the left and right displays need not be separate display screens, as they could instead be the left and right halves of a single display screen. The displays may be implemented using technology that provides sufficient display resolution or pixel density, e.g., liquid cry stal displaytechnology, organic light emitting diode technology, etc. Although not shown, there may also be an eyecup over each of the left and right displays that includes optics (e.g., a lens) serving to give the user the illusion that an object they see in the display (in this example a pine tree, which may be displayed in 2D or in 3D) is at a greater distance than the actual distance from their eye to the display, thereby enabling more comfortableviewing. The VR headset 1 might also incorporate trial lenses or some other adjustable refractive optical system to accommodate patients with different refractive errors.

[0018] The VR headset 1 also has a non- visible light-based eye tracking subsystem 8, e.g., an infrared or near infrared pupil tracking subsystem, whose output eye tracking data can be interpreted by the processor for independently tracking the positions of the left and right eyes, and for detecting blinks and pupil size or diameter of each eye, in a way that is invisible to the user. In one aspect, the eye tracking subsystem 8 is an infrared pupil tracking subsystem that produces images of pupils of the left eye and the right eye. The eye tracking subsystem 8 in that case may image the entirety of the left eye and the entirety of the right eye, and wherein the processor determines gaze angles of the left eye and the right eye based on: knowledge of distance between the right visible light display and the left visible light display; distance between the right eye and the right visible display; and location of a left pupil within the left eye and a right pupil within the right eye. or interpupillary distance.

[0019] In another aspect, the VR headset 1 has one or more light sensors that can be used to detect levels of light inside the left compartment and the right compartment, and the processor is configured to record the levels of light for the left compartment and the right compartment representing external light contribution while the user is wearing the VR headset 1. To avoid affecting the results of the test in an unrepeatable manner, and thereby make the test more reliable, the processor controls a parameter of the display (the left display 3 or the right display 4) to ensure that lighting in the compartment (the left compartment 5 or the right compartment 6, respectively), or chromaticity of the display, is consistent each time the vision test is conducted. The parameter is dependent on a color palette of the display and the nature of the lighting in the compartment.

[0020] In addition to the eye tracking subsystem 8, there may be a front facing internal camera, referred to as an on-axis camera in Fig. 3, that may be capable of capturing infrared or visible light or both. There is a separate front facing or on-axis camera for each of the left eye and the right eye (within their respective compartments 5, 6). The front facing or on-axis camera images the eye so that the processor can analyze its images to compute various metrics relevant to TED, as discussed below.

[0021] The system has a processor, e.g., one or more microelectronic processors that are part of the external computing device 9. one or more that are within the housing of the VR headset 1, or a combination of processors in those devices that are communicating with each other through a communication network interface. The processor is configured by software, or instructions stored in a machine readable medium such as solid state memory, to conduct a vision test, when the headset has been fitted over the user’s eyes. To do so, the processor signals the left or right visible light display to display a stimulus for the vision test that the user sees using their left or right eye, respectively. The processor may be configured to signal a further display, for example the display screen of the external computing device 9, to display progress of or the results of the test.

[0022] The TED vision test may proceed as follows, referring now to Fig. 2. The user is instructed, for example directly by an eye care professional, ECP, in-person or via previously recorded instructions that are played back through a speaker (of the headset for example), to fit the VR headset 1 over their eyes and look for a stimulus that will be displayed by the left visible light display or the right visible light display. As shown in the flow diagram of Fig. 2, the processor may then begin the TED vision test with operation 13. signaling the left or right visible light display to display stimuli for and then complete an initial subset (at least one) of the following vision tests (also referred to here as sub-tests):

[0023] a color vision test where the user’s eyes are employed by the system as a cursor or indicator, or the user manipulates a separate handheld controller, to complete for instance an Ishihara or D15 color test, e.g., as described in US patent application no. 18 / 530,062 filed Dec 5, 2023 (P1160US) which is incorporated by reference herein;

[0024] a contrast sensitivity test where the system employs the user’s eyes as a cursor or indicator, or responds to the user’s manipulation of the separate handheld controller, to complete a Pelli Robston test;

[0025] a visual acuity test where the user’s eyes as a cursor or pointer or the separate handheld controller is employed by the system to complete a visual acuity test, e g., as described in US patent application 18 / 193,193 filed Mar 30, 2023 (Pl 100US) which is incorporated by reference herein;

[0026] a pupil response test where the user would be instructed to stare ahead while their pupil size in each eye is recorded as each eye is independently stimulated, e.g., flashed, with a bright screen then kept dark and then repeated, e.g., as described in US patent application 18 / 447,936 filed Aug 10, 2023 (Pl 130US) which is incorporated by reference herein;

[0027] an eye motility test where the user would be instructed to look at various directions as far as they could go in each direction while their eye position is being recorded;

[0028] an eye alignment test where the user would be asked to stare at various fixation points throughout the field of view, the fixation points could either be presented to one or both eyes simultaneously, while the eyes are tracked to determine their positions regardless of whether they are presented stimuli, as for example described in US patent application 18 / 408,420 filed Jan 9, 2024 (P1140US) which is incorporated by reference herein;

[0029] a visual field test where the user would use either the separate handheld controller or their eye movements to point to various stimuli that are being presented to them in the VR headset, e g., as described in US patent application 18 / 537,645 filed Dec 12, 2023 (P1150US) which is incorporated by reference herein; and

[0030] ocular muscle testing in which the user is instructed to follow a stimulus that is on the center of their horizontal field of view, as the stimulus moves from the superior field of view to the inferior field of view.

[0031] In addition to performing one or more of the vision sub-tests listed above, the VR headset-based system for TED vision testing also determines and tracks other metrics in operation 15, that may be displayed to the ECP (see Fig. 1) to help the ECP diagnose a dry eye condition and / or TED. These metrics include blink rate and variance in blink rate, duration of blinks and variation in the duration, and the presence of incomplete blinks. Note that the operations described here need not be performed sequentially. For example, image data used to compute some of the metrics in operation 15 may be generated simultaneously with one of the vision sub-tests performed in operation 13.

[0032] In one aspect, digital images of the eye are captured by one or more internal cameras in the headset and then analyzed by the processor to determine a proptosis metric. Fig. 4 shows an example of images taken of the eyes of two subjects, by an optional side view internal camera in the headset. Fig. 2 shows an example orientation of the side view internal camera inside the headset, along with an on-axis camera, and another optional internal camera referred to as a below camera. The side-view camera is in front of a display (the left visible light display 3, or the right visible light display 4) with its imaging axis in this case perpendicular to that of the on-axis camera which is behind the display. There may also be a below camera whose imaging axis is angled upwards towards the eye. The side camera view may be used primarily for proptosis measurements as shown in Fig. 4. The side view camera may be a visible light camera that could also capture color images which the processor can analyze to determine eye redness (injection). The below camera may improve the eye tracking function especially when the user is looking down.

[0033] In another aspect, illumination within each of the left compartment 5 and the right compartment 6 of the headset is tuned to obtain color images captured by one or more of the internal cameras (see Fig. 3) that the processor analyzes to determine a metric (e.g., a chemosis metric, for example). The camera used for eye tracking (as part of the eye tracking subsystem 8) may only image in the infrared or near infrared, as visible light may be blocked from its imaging sensor with suitable optical filters. In one aspect, an optical filter for one or more of the internal cameras, for example the front facing or on-axis camera, is tuned to allow through some amount of red, green, and blue (or other primary color combination in the visible spectrum). The illuminators in each compartment of the headset would be similarly tuned and then turned on in sequence (e.g., red, then green, and then blue, or in another order) resulting in three monochrome images being captured by the internal camera. The processor would then merge these to form an RGB image, which is then processed to compute a color-based metric. The optical filter of the internal camera may also be designed to have notches that would block the visible light that is being emitted from or emerging from the display but would allow the light from the monochrome illuminators to pass through.

[0034] The processor may also determine additional metrics (operation 17), such as for lid lag (von Graefe sign), lid retraction, and scleral show. These metrics may becomputed by algorithmically segmenting the images of the eye (taken by one of the cameras in Fig. 3) to identify iris position, palpebral aperture (between upper and lower lid positions), and eyelid positions for both eyes. Fig. 5 shows how some of these metrics may be measured.

[0035] Once the metrics have been acquired (operation 15 and operation 17), they can be input to a trained machine learning model, ML model, that is trained to classify its input into one of several TED stages (e.g., no TED, early TED, mid stage TED, and late stage TED). The input to the ML model may also include the results of the initial subset, which is a subset of the vision sub-tests listed above that were completed upon the user in operation 13. If, at that point, a reasonable confidence level (above a threshold) is obtained for the output classification produced by the ML model, then the TED vision testing session may be deemed completed or finished. But if the confidence level is not sufficiently high (below the threshold) or if a higher quality baseline is desired for future progression monitoring of the TED, then a supplemental subset (one or more of the vision sub-tests listed above) are performed upon the user while the user is still wearing the headset following the administration of the initial subset. This is referred to in the figure as operation 18. The supplemental subset of vision sub-tests may include one or more of the vision sub-tests listed above that were not performed (as part of the initial subset). The supplemental subset may include all remaining ones in the list of sub-tests that have not yet been performed. The ML model may have been trained to classify its input based on not just the metrics mentioned above but also based on the results from all of the sub-tests in the list.

[0036] In another aspect, the user may be asked to return sometime later for another TED vision testing session, perhaps after a treatment was started following an initial diagnosis of the TED. The system and process described above may be repeated for subsequent TED vision testing sessions of the user. The results of each testing session may include the ML model’s output classification and the individual metrics mentioned above. These results can be logged by the system which the ECP uses to monitor the progression of the disease. The progression can be monitored by either the classification of the user, or by the individual metrics.

[0037] Referring now to Fig. 6, this figure illustrates another aspect of the disclosure here in which a smartphone-based TED screening app is developed thatwould bypass the headset (if for example ease of access is more important than accuracy). The figure suggests a timeline of repeated TED vision tests, performed by the app and by the headset-based system described above, where the instances of the TED vision test performed by the app are more frequent than the instances of the TED vision test performed by the headset-based system. The app could guide the user to move the smartphone camera around their head while capturing images that could by the app to measure proptosis, for example, or other metrics. The app may also perform one or more of the list of headset-based vision sub-tests described above upon their tested eye, as long as the user has some way to simultaneously block their untested eye (e.g., a patch). The results of the sub-tests and the metrics determined from the app and those from the headset-based vision system could be sent over the Internet to a remote computing process (e.g., in a cloud based service), where they are synchronized, to generate a unified picture of the progression (over time) of the user’s TED. Algorithmic analysis of such mixed progression data could weigh headset-based test results more strongly than smartphone-based test results.

[0038] While certain aspects have been described and shown in the accompanying drawings, it is to be understood that such are merely illustrative of and not restrictive on the broad invention, and that the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those of ordinary skill in the art. For instance, while the TED screening app is referred to above as being smartphone-based, the app could perform similarly even if it were designed to be executed in another type of a user portable handheld device that has a built-in camera, such as a tablet computer. The description is thus to be regarded as illustrative instead of limiting.

Claims

CLAIMSWhat is claimed is:

1. A thyroid eye disease, TED, testing system comprising: a VR headset; and a processor configured to, when the headset has been fitted over the user’s eyes, i) perform an initial subset of one or more of a plurality of vision sub-tests to produce results, by a) signaling the VR headset to display stimuli that the user can see and b) analyzing tracking data from an eye tracking subsystem of the VR headset and / or image data from one or more internal cameras of the VR headset that capture the user’s eyes while the stimuli the stimuli are displayed in a). ii) determine a plurality of metrics relevant to dry eye diagnosis or TED diagnosis by processing the image data from the internal cameras of the VR headset, and iii) determine a classification of TED stage of the user based on the plurality of metrics and the results of the initial subset.

2. The system of claim 1 wherein the processor is further configured to display the classification on an external device.

3. The system of any one of the previous claims wherein the processor is configured to: determine a confidence level for the classification; and indicate that a TED vision testing session is finished and store the classification and the plurality of metrics as being associated with the TED vision testing session, in response to the confidence level being greater than a threshold.

4. The system of any one of claims 1-2 wherein the processor is configured to: determine a confidence level for the classification; and in response to the confidence level being lower than a threshold, perform a supplemental subset of one or more of the plurality of vision sub-tests to produce supplemental results, by a) signaling the VR headset to display stimuli for the supplemental subset and b) analyzing tracking data from the eye tracking subsystem ofthe VR headset and / or image data from the one or more internal cameras of the VR headset that capture the user’s eyes while the stimuli for the supplemental subset are displayed in a).

5. The system of any one of claims 1-2 wherein the processor is configured to: in response to a higher quality baseline being desired for future progression monitoring of TED, perform a supplemental subset of one or more of the plurality of vision sub-tests to produce supplemental results, by a) signaling the VR headset to display stimuli for the supplemental subset and b) analyzing tracking data from the eye tracking subsystem of the VR headset and / or image data from the one or more internal cameras of the VR headset that capture the user’s eyes while the stimuli for the supplemental subset are displayed in a).

6. The system of any one of the previous claims wherein the plurality of vision sub-tests is two or more of the following: a color vision test; a contrast sensitivity test; a visual acuity test; a pupil response test; an eye motility' test; an eye alignment test; a visual field test; and ocular muscle testing.

7. The system of any one of the previous claims wherein the VR headset comprises: a left visible light display, a left compartment to fit over a left eye of a user, a right visible light display, a right compartment to fit over a right eye of the user, wherein the left and right compartments are configured so that when the headset has been fitted over the user’s eyes a) the user cannot see the right display using only their left eye and b) the user cannot see the left display using only their right eye, wherein the eye tracking subsystem that produces tracking data for the left eye and for the right eye is non-visible light-based.

8. The system of claim 7 wherein the processor controls a parameter of the left or right display to ensure that lighting in the left or right compartment, or chromaticity of the left or right display, do not affect results of the test in an unrepeatable manner9. The system of any one of the previous claims wherein the processor is external to the VR headset, and the VR headset comprises a wired or wireless communications network interface through which the tracking data from the eye tracking subsystem is sent to the processor.

10. An article of manufacture comprising: a machine readable medium having instructions stored therein that configure a processor in a smartphone or a tablet computer to: guide a user of the smartphone or tablet computer to aim a camera of the smartphone or tablet computer at their eye or head, while the camera is capturing images of the user’s eye; determine one or more metrics relevant to dry eye diagnosis or TED diagnosis by processing the images; perform a subset of one or more of a plurality of vision sub-tests to produce test results, by i) signaling a display of the smartphone or tablet computer to display stimuli that the user can see and ii) analyzing images from the camera that capture the user’s eye while the stimuli are being displayed in i); and send the test results and the one or metrics, along with i) a date stamp and / or a time stamp for when the test results were produced or when the one or metrics were determined and ii) data that associates the user with the test results and the one or more metrics, over the Internet to a remote computing process.

11. The article of manufacture of claim 10 wherein the processor is configured to determine a classification of TED stage of the user based on the one or more metrics and the test results, and send the classification over the Internet to the remote computing process.

12. The article of manufacture of claim 10 or 11 wherein the remote computing process receives a plurality of results and a plurality of metrics, along with a data stamp and / or a time stamp for when a VR headset-based system has produced the plurality of test results and the plurality of metrics, from the VR headset-based system, and in response synchronizes i) the plurality of results and the plurality of metrics received from the VR headset-based system with ii) the test results and the one or metrics received from the smartphone or the tablet computer, to generate a unified progression of the user’s TED.

Images