Systems and methods for capturing and rendering depth information in an eye

Abstract

A system and method for capturing and rendering depth information in an eye, comprising illuminating via a scanning-based instrument an eye structure from a first off-center position, illuminating via the scanning-based instrument the eye structure from at least one second off-center position different from the first, capturing an image of the illuminations in a multiple perspective fashion, and rendering depth information based on the captured image of the illuminations.

Description

SYSTEMS AND METHODS FOR CAPTURING AND RENDERING DEPTH INFORMATION IN AN EYECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. provisional application No. 63 / 642,053 filed on May 3, 2024, incorporated herein by reference in its entirety.STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] This invention was made with government support under EY001319, and EY028293 awarded by the National Institutes of Health. The government has certain rights in the invention.BACKGROUND OF THE INVENTION

[0003] High resolution imaging of the eye is typically achieved by using the full pupil of the eye in order to maximize the numerical aperture. For example, in the field of adaptive optics ophthalmoscopy, which seeks to maximize lateral and axial resolution of the target object, it is common practice to illuminate the full aperture of the eye (typically the pupil). In the field of confocal imaging, a physical or synthesized pinhole is placed conjugate to the focal plane of illumination. This provides sharp images of the target focal plane, but limits useful information from above or below the focal plane, thus destroying, confounding or rejecting useful depth information.

[0004] Thus there is a need in the art for improved systems and methods for capturing and rendering depth information in an eye.SUMMARY OF THE INVENTION

[0005] Some embodiments of the invention disclosed herein are set forth below, and any combination of these embodiments (or portions thereof) may be made to define another embodiment.

[0006] In one aspect, a method for capturing and rendering depth information in an eye comprises illuminating via a scanning-based instrument an eye structure from a first off-center position, illuminating via the scanning-based instrument the eye structure from at least one second off-center position different from the first, capturing an image of the illuminations in a multiple perspective fashion, and rendering depth information based on the captured image of the illuminations.

[0007] In one embodiment, the scanning-based instrument comprises an adaptive optics ophthalmoscope.

[0008] In one embodiment, the image of the illuminations is provided to a user in a binocular fashion thus providing for rendering of depth information to the user naturally via stereopsis.

[0009] In one embodiment, the image of the illuminations is provided to a computing system.

[0010] In one embodiment, the computing system renders the depth information.

[0011] In one embodiment, the method is performed in real-time.

[0012] In one embodiment, the method further comprises tracking an object moving into or out of a plane of focus relative to the illuminations.

[0013] In one embodiment, the illuminations provide a parallax view of the eye structure.

[0014] In one embodiment, the method further comprises performing multimodal capture comprising at least one of reflectance, phase contrast, absorption, polarization, birefringent,and spectral and fluorescence modalities including one-photon and multiple photon fluorescence.

[0015] In one embodiment, the illuminations comprise light of different polarizations.

[0016] In one embodiment, the illuminations comprise light of different wavelengths.

[0017] In one embodiment, the light of different wavelengths is configured to excite one or more fluorophores.

[0018] In one embodiment, the illuminations are applied simultaneously.

[0019] In one embodiment, the illuminations are applied sequentially to create frames, lines, and / or points.

[0020] In one embodiment, the frames, lines, and / or points are interleaved to provide temporal multiplexing.

[0021] In one embodiment, wherein one illumination is presented in a forward scan and the other illumination is presented in a backward scan of an ophthalmoscope.

[0022] In one embodiment, the depth information is rendered by way of binocular disparity via at least one of a chromatic anaglyph, a polarization goggle, an augmented virtual display, a lenticular display, and an alternating temporal shutter.

[0023] In another aspect, a device for capturing and rendering depth information in an eye, comprises a light source, one or more scanning optical elements configured to direct light from the light source to provide multiple illuminations of an eye structure from at least two off- center positions relative to a pupillary axis, and a capture sensor configured to capture an image of the multiple illuminations in a multiple perspective fashion.

[0024] In one embodiment, the device comprises means for moving the device relative to the eye.

[0025] In one embodiment, the device comprises means for moving the eye relative to the device.

[0026] In another aspect, a system for capturing and rendering depth information in an eye comprises an ophthalmoscope configured to perform the method as describe above.

[0027] In one embodiment, the system further comprises a computing system communicatively connected to the ophthalmoscope, comprising a processor and a non- transitory computer-readable medium with instructions stored thereon, which when executed by the processor, perform steps comprising rendering depth information based on the captured image of the illuminations,

[0028] In one embodiment, a full or near full pupil of the eye is utilized for adaptive optics wavefront sensing to provide aberration sensing / correction using a spectrum or a wavelength of light.

[0029] In one embodiment, the system is configured to provide at least one of a second wavelength, a polarization multiplexed, and a time multiplexed illumination beam projected simultaneously or sequentially onto a fraction of the pupil to perform decentered axis illumination of the sample.BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The foregoing purposes and features, as well as other purposes and features, will become apparent with reference to the description and accompanying figures below, which are included to provide an understanding of the invention and constitute a part of the specification, in which like numerals represent like elements, and in which:

[0031] Figs. 1A and IB depict a comparison of conventional adaptive optics ophthalmoscopy (Fig. 1A) to an off-center ophthalmoscopy method of the present invention (Fig. IB) in accordance with some embodiments.

[0032] Fig. 2 depicts a comparison of illumination point spread functions for a fully illuminated pupil (left) to off-center partially illuminated pupils (center, right) in accordance with some embodiments.

[0033] Figs. 3A to 3C depict a comparison of perspectives of an observer of conventional full illumination of a pupil (Fig. 3A) to off-center partially illuminated pupils (Figs. 3B-3C) in accordance with some embodiments.

[0034] Fig. 4 depicts simulations of pupil illumination conditions and recovery of depth information in accordance with some embodiments.

[0035] Fig. 5 depicts free fusion of two experimental perspective images to create stereodepth information in accordance with some embodiments.

[0036] Fig. 6 depicts offset illuminations of a circular pupil to create experimental perspective images which provide a parallax view rendering depth information in accordance with some embodiments.

[0037] Fig. 7 depicts chromatic anaglyphs in accordance with some embodiments.

[0038] Fig. 8 depicts exemplary experimental retinal images over time produced via the disclosed systems and methods in accordance with some embodiments.

[0039] Fig. 9 depicts a comparison of experimental left and right stereo views (top row) and left and right non-stereo views (bottom row) in accordance with some embodiments.

[0040] Figs. 10A-10B depict a 3D phase space rendering of the stereo data in accordance with some embodiments.

[0041] Fig. 11 depicts an exemplary computing environment in which aspects of the invention may be practiced.DETAILED DESCRIPTION OF THE INVENTION

[0042] It is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clearer comprehension of the present invention, while eliminating, for the purpose of clarity, many other elements found in systems and methods for capturing and rendering depth information in an eye. Those of ordinary skill in the art may recognize that other elements and / or steps are desirable and / or required in implementing the present invention. However, because such elements and steps are well known in the art, and because they do not facilitate a better understanding of the present invention, a discussion of such elements and steps is not provided herein. The disclosure herein is directed to all such variations and modifications to such elements and methods known to those skilled in the art.

[0043] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, exemplary methods and materials are described.

[0044] As used herein, each of the following terms has the meaning associated with it in this section.

[0045] The articles "a" and "an" are used herein to refer to one or to more than one ( / .e., to at least one) of the grammatical object of the article. By way of example, "an element" means one element or more than one element.

[0046] "About" as used herein when referring to a measurable value such as an amount, a temporal duration, and the like, is meant to encompass variations of ±20%, ±10%, ±5%, ±1%, and ±0.1% from the specified value, as such variations are appropriate.

[0047] Ranges: throughout this disclosure, various aspects of the invention can be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation onthe scope of the invention. Where appropriate, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 2.7, 3, 4, 5, 5.3, and 6. This applies regardless of the breadth of the range.

[0048] Referring now in detail to the drawings, in which like reference numerals indicate like parts or elements throughout the several views, in various embodiments, presented herein are systems and methods for capturing and rendering depth information in an eye.

[0049] In some embodiments, the disclose systems and methods provide depth information to the user / imager / surgeon by providing multiple perspectives of the same target structures through a scanning system such as an adaptive optics ophthalmoscope.

[0050] Figs. 1A and IB depict a comparison of conventional adaptive optics ophthalmoscopy (Fig. 1A) to an off-center ophthalmoscopy method of the present invention (Fig. IB). Fig. 1A shows the conventional way that adaptive optics ophthalmoscopy is performed. The full pupil is illuminated to maximize numerical aperture and maximize lateral and axial resolution of the system. Fig. IB shows how the disclosed systems and methods displace the illumination of the eye (such as with a laser ophthalmoscope or similar) to illuminate the structure from an off-center position. The combination of one or more of these lateralized illumination configurations allows for measurements of depth by virtue of triangulation. By illuminating one or more lateralized locations of the pupil a non-symmetrical point spread function relative to the principal axis of the eye is created.

[0051] Fig. 2 depicts a comparison of illumination point spread functions for a fully illuminated pupil (left) to off-center partially illuminated pupils (center, right). Maximizing the entrance pupil for imaging (left) creates an illumination point spread function that is narrow at the focal plane (narrow depth of field). Displacing the illumination of the eye to a fraction of the full pupil (center, right) provides for one or more illumination angles (i.e. one or moreperspective angles) of the same imaged target which creates perspective or parallax imaging. Depth is thus recoverable from these images by a computer, or with the human visual system by presenting one or more vantage points to the human observer (imaging operator).

[0052] For example, when illuminating the left portion of the pupil only, the illumination axis is tilted away from the principal axis of the eye. This angled illumination creates a unique perspective of the target object imaged. By combining one or more unique angles multiple perspectives of the same object are created which creates depth, tomography, and / or ranging information. Thus, adaptive optics imaging can be accompanied by non-symmetrical illumination configurations to create multiple vantage points that can be achieved simultaneously, sequentially, or by using a number of properties of light that allow multiplexing of different perspective angles simultaneously and / or in rapid temporal succession.

[0053] Furthermore, in some embodiments, computational and / or rendering math is not required in order to re-create depth for the user / observer. For example, a user or operator (for example a surgeon) who views such an image in a binocular fashion, each illumination perspective produces an image of the same structure from two different vantage points, thus creating stereopsis. Therefore a major advantage of the disclosed systems and methods are that in some embodiments no additional computers, algorithms, and / or complicated math is required to render depth information. Instead, information from the multiple perspective angles can be presented to two eyes of the user simultaneously. This makes use of the stereopsis, disparity and binocular system of the human visual system to serve as the "computer" to render depth information naturally. In this way, the human viewer becomes the complicated computer that renders such information useful. Further, the depth information can thus be rendered in real-time.

[0054] Figs. 3A to 3C depict a comparison of perspectives of an observer of conventional full illumination of a pupil (Fig. 3A) to off-center partially illuminated pupils (Figs. 3B-3C). Fig. 3A shows conventional illumination using a full pupil. Object B is in the focus plane. Object A is above and C is below the focal plane (middle panel). Due to the symmetrical nature of imaging in this way, the imaged view of the scene shows a lateral super-position of A, B, C (bottompanel). However, imaging with left perspective (Fig. 3B) produces an offset view of scene A,B,C which renders a lateral displacement of the structures. Deep objects (C) are represented to the left of the object at the focal plane (B). Objects above the focal plane (A) are represented to the right of the object at the focal plane (B). Fig. 3C shows the perspective of the same objects reverses the appearance of "ABC" when imaged with right perspective. Thus, the lateralization of the illumination perspective changes the appearance of the object to give multiple perspectives. The shift (left-right or right-left) reveals whether the object is above or below the focal plane and the magnitude of the shift reveals the distance above or below the focal plane (ranging).

[0055] With reference to Fig. IB, in one embodiment, a method for capturing and rendering depth information in an eye comprises illuminating via a scanning-based instrument 100 an eye structure from a first off-center position 106, illuminating via the scanning-based instrument the eye structure from at least one second off-center position different from the first, capturing an image of the illuminations in a multiple perspective fashion, and rendering depth information based on the captured image of the illuminations. In one embodiment, a single scanning-based instrument is utilized to provide the illuminations. In one embodiment, two or more scanning-based instruments are utilized to provide the illuminations. In one embodiment, the scanning-based instrument 100 comprises an adaptive optics ophthalmoscope, a scanning light ophthalmoscope, a line scanning ophthalmoscope, a flood illumination adaptive optics ophthalmoscope, or any other suitable scanning-based instrument or combinations thereof.

[0056] In one embodiment, the image of the illuminations is provided to a user in a binocular fashion thus providing for rendering of depth information to the user naturally via stereopsis. In one embodiment, the illuminations provide a parallax view of the eye structure. In one embodiment, the method is performed in real-time or near real time. In one embodiment, the method is performed in less than 5 seconds, less than 1 second, less than 0.1 second, less than 0.01 sec, less than 103sec, less than 105sec, less than 109sec, less than 1012sec, or less than 1015sec. In one embodiment, the depth information is rendered by way ofbinocular disparity via at least one of a chromatic anaglyph, a polarization goggle, an augmented reality display, a virtual display, a lenticular display, and / or an alternating temporal shutter.

[0057] In one embodiment, the image of the illuminations is optionally provided to a computing system 105. In one embodiment, the computing system 105 renders the depth information via any suitable methods, such as, but not limited to, 3D rendering where the display rotates perspectives in time (rotating cube), chromatic channels such as an anaglyph, polarization information presented to the user, virtual or augmented goggles, and lenticular displays.

[0058] In one embodiment, the method further comprises tracking an object moving into or out of a plane of focus relative to the illuminations.

[0059] In one embodiment, the method further comprises, via a capture sensor 103, performing multimodal capture comprising at least one of reflectance, phase contrast, absorption, polarization, birefringent, and spectral and fluorescence modalities including one- photon and multiple photon fluorescence.

[0060] In one embodiment, the illuminations comprise light of different polarizations. In one embodiment, the illuminations comprise light of different wavelengths. In one embodiment, the light of different wavelengths is configured to excite one or more fluorophores. In one embodiment, the illuminations are applied simultaneously. In one embodiment, the illuminations are applied sequentially to create frames. In one embodiment, the frames are interleaved to provide temporal multiplexing. In one embodiment, one illumination is presented in a forward scan and the other illumination is presented in a backward scan of an ophthalmoscope.

[0061] With reference to Fig. IB, in one embodiment, a device 100 for capturing and rendering depth information in an eye comprises a light source 101, one or more scanning optical elements 102 configured to direct light from the light source 101 to provide multiple illuminations of an eye structure from at least two off-center positions 106 relative to apupillary axis, and a capture sensor 103 configured to capture an image of the multiple illuminations in a multiple perspective fashion. In one embodiment, the device 100 comprises a positioning mechanism 104 for moving the device relative to the eye, such as pneumatic actuators, linear actuators, lead screws, belts, pulleys, electromagnetic actuators, optical translation or angular deflection of mirrors or prisms. In some embodiments, the positioning mechanism 104 includes an automation system for moving the device automatically relative to the eye, for example according to a pre-set routine. In other embodiments, the positioning mechanism 104 may comprise the device being manually moved or positioned by a user (human), or similar. 11 n one embodiment, the device comprises a positioning mechanism 104 for moving the head and / or the eye, relative to the device, such as pneumatic actuators, linear actuators, lead screws, belts, pulleys, electromagnetic actuators, displaying a target, an instructional cue for the subject or patient to manually change their gaze position relative to the static instrument, presenting a fixation or gaze direction target to the user through the instrument within the imaging environment or as a task-dependent command given to the subject, or similar. In one embodiment, the light source comprises a laser (Including continuous wave or pulsed sources), an incandescent bulb, a light emitting diode, or any other suitable light source or combinations thereof.

[0062] In another aspect, a system for capturing and rendering depth information in an eye comprises an ophthalmoscope configured to perform the method as described above.

[0063] In one embodiment, the system further comprises a computing system 105 communicatively connected to the ophthalmoscope, comprising a processor and a non- transitory computer-readable medium with instructions stored thereon, which when executed by the processor, perform steps comprising rendering depth information based on the captured image of the illuminations.

[0064] In one embodiment, a full or near full pupil of the eye is utilized for adaptive optics wavefront sensing to provide aberration sensing / correction using one wavelength of light. In one embodiment, the system is configured to provide at least one of a second wavelength, apolarization multiplexed, and a time multiplexed illumination beam projected simultaneously or sequentially onto a fraction of the pupil to perform decentered axis illumination of the sample.

[0065] In one embodiment, the system and method provide real time stereo imaging with simultaneous capture, which can allow, for example, a clinician to be able to visualize single blood cells in slow moving vasculature.

[0066] In one embodiment, the system or method is configured to correct higher order aberrations.

[0067] In one embodiment, the system or method is configured to measure imperfections of the eye lens and correct said imperfections via a morphable mirror or similar.

[0068] Provided below are exemplary ways multiple illumination angles can be simultaneously or sequentially presented. The examples below use "left" and "right" illumination profiles, but in practice any angle, offset or combination of multiple perspectives can be used:1) Left and right simultaneously illuminated using polarized light of different linear polarization or circularly polarized light. Imaging detectors using parallel or orthogonal detection.2) Left and right simultaneously illuminated using light of differing wavelength, (e.g. left illuminated with 795 nm light, right illuminated 796 nm light) In such a convention, impact of longitudinal and transverse chromatic aberration are negligible, yet left and right parallax information may be chromatically multiplexed. Grayscale intensity images which do not provide color information are presented to the observer to allow for only depth information to be visualized without chromatic rivalry between the two eyes, such as with a chromatic anaglyph.3) Left and right sequentially presented. As in a video, frame 1 is illuminated by left, frame 2 is illuminated by right. Interleaved frames provide temporal multiplexing at frame-rate speeds.4) Left and right sequentially presented in forward and backward scans. In most scanning systems, light is propagated to the tissue during the forward and backward propagation of a mirror. In this configuration, the "left illumination" is presented in the forward scan, and "right illumination" is presented in the backward scan. This allows for line-level interleaving of left and right information which dramatically increases interleave speeds to that faster than the human visual system can detect, creating a stereoscopic image without perceptual flicker.5) Left and right information presented simultaneously or sequentially using two separate wavelengths to excite fluorescence of one or more fluorophores. Emitted fluorescence can be captured using one or more fluorescence emission filters.6) Any separable property of light (including but not limited to spectra, polarization, variable speed through media) can be used to create simultaneous or sequential multiplexing of two or more perspective angles.7) Any combination of the above examples with each other.

[0069] Fundamental differences from other tomography, ranging or depth information in ocular imaging devices is described below. While Optical Coherence Tomography and Adaptive Optics OCT have enabled depth resolution, they do so with two notable differences to the disclosed systems and methods, first both use time-domain, spectral-domain, Fourier-Domain, or swept-source domain tomography to render information regarding depth. This typically requires a computer and algorithms to render such data to the user to be interpreted. This delays information (e.g. information is not real-time, creating lag) and adds complexity and cost. The novel systems and methods disclosed herein only require a minimum of two illumination angles, creating a parallax view of the scene. This can be achieved without a computer to render depth information. Further, the human visual system (such as that of the surgeon) can serve as the "disparity / depth computer".

[0070] Moreover, tomography approaches like OCT and ultrasound do not allow for fluorescence detection. As described herein the disclosed configuration can be combined with fluorescence, thus enabling 3D information of fluorescence within the volumetric field.Therefore, fluorescent structures (e.g. cells or retinal vessels) can be appreciated in 3D without changing the focal plane or computing tomography. This allows real-time visualization (e.g. utility in surgery). Additionally, it allows for multimodal capture with reflectance, absorption, birefringent, spectral and fluorescence modalities can be combined into the same device to allow for superposition of these modalities all while providing depth information in all -or some- of the imaging channels.

[0071] The disclosed systems and methods can be utilized in a wide-ranging applications. For example, any scenario where a real-time display of 3D information is required from the eye / retina could benefit a user utilizing the disclosed systems and methods.. A non-exhaustive list of hypothetical examples are listed below to show scope and utility:

[0072] In one example, a surgeon requires real-time stereoscopic information for ocular surgery. General surgeons are known to use stereo-vision to aid in precise cutting, manipulation, and / or resection of tissue. In the case of ocular surgeons that must contend with the eye and retina, even more detailed precision is needed. For example, ocular and retinal surgeons often work with tissue 10-100s of micrometers thick. When performing procedures like an inner limiting membrane peel (a tissue is about 10 micrometers thick) it is common practice to remove deleterious effects of retinal traction and macular hole in retinal complications. Being able to have micron-level surgical precision of both surgical instruments and their position relative to tissue provides a surgical advantage. Furthermore, as the information can be presented in real-time and without requiring computer rendering, the surgeons can operate with exquisite stereo-depth information combined with natural proprioception and haptic feedback of the surgical field without multisensory delay.

[0073] In another example, retinal surgeons currently perform delicate surgery on retinal tissue that is 100-300 micrometers thick. Being able to place bionic retinal implants, delivery of drugs, foreign objects, and / or viral-vector payloads in sub-retinal injections or other detailed surgical interventions requiring microscopic precision would benefit from real-time visualization of ocular tissue, surgical field, and surgical instruments.

[0074] In another example, in the domain of research, being able to watch and or track objects as they move in and out of the plane of focus would render depth information to cells, sub-cellular structures, and microscopic objects that move within the plane of the depth of field which may span microns to millimeters.

[0075] In yet another example, combining the disclosed approach with fluorescence, when the ocular field is illuminated by two point spread functions from lateralized entry points in the pupil it produces two or more perspectives of a fluorescent target object (as well as those above and below the plane of optimum focus). This allows for 3D ranging of fluorescent objects. In one example, fluorescent vessels labeled with fluorescein may be visualized and / or rendered in 3D by comparing at least two perspective angles. Fluorescent objects at the plane of focus show zero shift between the two perspective images. Vessels above the plane show a double image with the magnitude of depth information encoded by the degree of shift between the two perspective images. Vessels below the plane show the same magnitude depth effect, but the vector of shift between the two objects is reversed (See Fig. 3A, Fig. 3B, and Fig. 3C). The fluorescence depth information can thus be captured with "single shot" imaging from two cameras. This differs substantially from the way fluorescence information is captured in an ophthalmoscope currently which captures the fluorescence information by progressively changing the focal plane and capturing a new image. This is conventionally referred to as a "z- stack" or a progressive, serial capture of multiple planes over time. This is especially problematic in live retinal imaging as the eye is constantly in motion, meaning multiple planes may be sheared relative to one another. Progressive plane imaging is also problematic because it is expensive, slow and may use all of the allowable "light-budget" to image the retina in an incomplete subset of the entire z-stack (a common problem in ocular imaging). Similarly, progressive imaging may be difficult to render in situations where the fluorophore is bleached with progressive duration of light. The disclosed systems and methods can capture all depth information at a single point in time or two or more images captured in rapid succession by comparing two or more perspectives. In the context of application, a surgeon could therefore visualize fluorescence depth information in real time to guide live-surgery.

[0076] In a final example, in the domain of imaging system optimization, having two illumination beams from opposite and symmetrical sides of the pupil allows one to identify where the center of the focal plane is in the axial dimension. For example one could use the dual off-center illumination approach to zero-in on the plane at which there is no "doubleimage". This would define the focal plane of the illumination, which is typically assessed by confocal strategies.

[0077] In summary, utilization of the disclosed systems and methods enables one to see ocular tissue in 3D. Using the natural viewing system of human eyes (as conventionally thought of in binoculars), one can visualize the retina using adaptive optics in stereo, a rich 3D environment that humans are accustomed to using in their natural environments. In practical and commercial applications, the disclosed systems and methods have numerous uses, including but not limited to:1) Intraoperative surgery. Surgeons utilize stereo viewing systems for surgical intervention. Combined with adaptive optics, such a system would allow micron- nanometer level resolution in not only the lateral (en face dimension), but also with comparable detail in the axial dimension. Such information is useful for surgeons which must contend with complex 3D tissue for interaction (e.g. Tumor or tissue resection) and viewing surgical instruments in relation to that tissue in real-time.2) Depth ranging. By using multiple perspective imaging with adaptive optics, one can render epi-illuminated targets with information of what objects are above and which are behind the plane of best focus. Objects in front of the focal plane will parallax shift in one direction. Objects behind the focal plane will parallax shift in an opposite direction (Fig. 3A, Fig. 3B, and Fig. 3C). Objects in plane will not shift at all.3) Using the parallax observations, this permits fine tuning of the focal plane to the object of interest without using a confocal pinhole which not only rejects light, but largely removes information about the polarity of focus (above or below plane).EXPERIMENTAL EXAMPLES

[0078] The invention is now described with reference to the following Examples. These Examples are provided for the purpose of illustration only and the invention should in no way be construed as being limited to these Examples, but rather should be construed to encompass any and all variations which become evident as a result of the teaching provided herein.

[0079] Without further description, it is believed that one of ordinary skill in the art can, using the preceding description and the following illustrative examples, make and utilize the present invention and practice the claimed methods. The following working examples therefore, specifically point out exemplary embodiments of the present invention, and are not to be construed as limiting in any way the remainder of the disclosure.

[0080] Fig. 4 depicts simulations of three pupil illumination conditions. Full pupil aperture (top) creates a cone of light with a focus at a tight plane representing a narrow depth of field (middle panel). The enface view reveals a small, symmetrical point spread function (right panel). The right-illumination condition creates a larger axial-depth of field (middle panel). When color coding the point spread function for depth, blue-green colors are above the plane of focus and orange-red colors are below the plane of focus. The opposite is true for leftillumination despite having a point spread function of nearly equivalent size.

[0081] For viewing, images may be combined using "free fusion" by crossing one's eyes to combine left and right eye information simultaneously. Fig. 5 depicts free fusion from two rapidly captured images using left and right illumination angles creates stereo-depth information.

[0082] Fig. 6 depicts offset illuminations of a circular pupil to create offset images which provide a parallax view rendering depth information. Left illumination of a circular pupil (top left) creates image at left. Right illumination creates image at right. A combination of the two images creates a parallax view of the same scene, thus rendering depth.

[0083] Fig. 7 depicts chromatic anaglyphs that combines left (pseudo colored red) and right (pseudo colored cyan) perspectives of retinal tissue or tissue phantom captured in anadaptive optics scanning light ophthalmoscope. When viewed with "3D anaglyph glasses" the left and right eye of the observer sees two perspectives of the same image encoded by color. Depth information is rendered to the observer by nature of computations performed in the human visual system (visual cortex). The left ocular uses a red filter while the right ocular uses a cyan filter. The top image shows an In vivo retinal image of mouse retina., the middle Image shows an image of a model retina, and the bottom image shows a model retina with sparse fibers.

[0084] Fig. 8 depicts exemplary experimental still images over time (left to right, top to bottom) from a captured video file produced via the disclosed systems and methods which shows how exemplary stereo information can be captured with adaptive optics configurations. The experimental images show details of vessels and blood cells moving through the vessel.

[0085] Fig. 9 depicts a proof-of-concept comparison of experimental left and right stereo views (top row) and left and right non-stereo views (bottom row) image of the vessels of the living eye captured by the systems and methods described herein. The stereo view requires free-fusing the two images to produce stereo / depth view of the vessels. The two perspectives captured either sequentially or simultaneously.

[0086] Figs. 10A-10B show 3D rendering of the tomography data, but now visualized by projecting the data into a 3D visualization. Fig. 10A shows a phase space of the tomography data, where the slope of the vessel in the scan direction is indicative of the vessel depth. The 0 dimension corresponds to displacement of the illumination slit in the x direction. The slope of each vessel in the 0 dimension represents its depth from the reference plane. Fig. 10B shows various views of the tomography data.COMPUTING ENVIRONMENT

[0087] In some aspects of the present invention, software executing the instructions provided herein may be stored on a non-transitory computer-readable medium, wherein thesoftware performs some or all of the steps of the present invention when executed on a processor.

[0088] Aspects of the invention relate to algorithms executed in computer software. Though certain embodiments may be described as written in particular programming languages, or executed on particular operating systems or computing platforms, it is understood that the system and method of the present invention is not limited to any particular computing language, platform, or combination thereof. Software executing the algorithms described herein may be written in any programming language known in the art, compiled or interpreted, including but not limited to C, C++, C#, Objective-C, Java, JavaScript, MATLAB, Python, PHP, Perl, Ruby, or Visual Basic. It is further understood that elements of the present invention may be executed on any acceptable computing platform, including but not limited to a server, a cloud instance, a workstation, a thin client, a mobile device, an embedded microcontroller, a television, or any other suitable computing device known in the art.

[0089] Parts of this invention are described as software running on a computing device. Though software described herein may be disclosed as operating on one particular computing device (e.g. a dedicated server or a workstation), it is understood in the art that software is intrinsically portable and that most software running on a dedicated server may also be run, for the purposes of the present invention, on any of a wide range of devices including desktop or mobile devices, laptops, tablets, smartphones, watches, wearable electronics or other wireless digita l / cel lula r phones, televisions, cloud instances, embedded microcontrollers, thin client devices, or any other suitable computing device known in the art.

[0090] Similarly, parts of this invention are described as communicating over a variety of wireless or wired computer networks. For the purposes of this invention, the words "network", "networked", and "networking" are understood to encompass wired Ethernet, fiber optic connections, wireless connections including any of the various 802.11 standards, cellular WAN infrastructures such as 3G, 4G / LTE, or 5G networks, Bluetooth®, Bluetooth® Low Energy (BLE) or Zigbee® communication links, or any other method by which one electronic device is capableof communicating with another. In some embodiments, elements of the networked portion of the invention may be implemented over a Virtual Private Network (VPN).

[0091] Fig. 11 and the following discussion are intended to provide a brief, general description of a suitable computing environment in which the invention may be implemented. While the invention is described above in the general context of program modules that execute in conjunction with an application program that runs on an operating system on a computer, those skilled in the art will recognize that the invention may also be implemented in combination with other program modules.

[0092] Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0093] Fig. 11 depicts an illustrative computer architecture for a computer 800 for practicing the various embodiments of the invention. The computer architecture shown in Fig. 11 illustrates a conventional personal computer, including a central processing unit 850 ("CPU"), a system memory 805, including a random-access memory 810 ("RAM") and a readonly memory ("ROM") 815, and a system bus 835 that couples the system memory 805 to the CPU 850. A basic input / output system containing the basic routines that help to transfer information between elements within the computer, such as during startup, is stored in the ROM 815. The computer 800 further includes a storage device 820 for storing an operating system 825, application / program 830, and data.

[0094] The storage device 820 is connected to the CPU 850 through a storage controller (not shown) connected to the bus 835. The storage device 820 and its associated computer-readable media, provide non-volatile storage for the computer 800. Although the description of computer-readable media contained herein refers to a storage device, such as a hard disk or CD-ROM drive, it should be appreciated by those skilled in the art that computer-readable media can be any available media that can be accessed by the computer 800.

[0095] By way of example, and not to be limiting, computer-readable media may comprise computer storage media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.

[0096] According to various embodiments of the invention, the computer 800 may operate in a networked environment using logical connections to remote computers through a network 840, such as TCP / IP network such as the Internet or an intranet. The computer 800 may connect to the network 840 through a network interface unit 845 connected to the bus 835. It should be appreciated that the network interface unit 845 may also be utilized to connect to other types of networks and remote computer systems.

[0097] The computer 800 may also include an input / output controller 855 for receiving and processing input from a number of input / output devices 860, including a keyboard, a mouse, a touchscreen, a camera, a microphone, a controller, a joystick, or other type of input device. Similarly, the input / output controller 855 may provide output to a display screen, a printer, a speaker, or other type of output device. The computer 800 can connect to the input / output device 860 via a wired connection including, but not limited to, fiber optic, ethernet, or copper wire or wireless means including, but not limited to, Bluetooth, Near-Field Communication (NFC), infrared, or other suitable wired or wireless connections.

[0098] As mentioned briefly above, a number of program modules and data files may be stored in the storage device 820 and RAM 810 of the computer 800, including an operating system 825 suitable for controlling the operation of a networked computer. The storage device 820 and RAM 810 may also store one or more applications / programs 830. In particular, the storage device 820 and RAM 810 may store an application / program 830 for providing a variety of functionalities to a user. For instance, the application / program 830 may comprise many types of programs such as a word processing application, a spreadsheet application, a desktop publishing application, a database application, a gaming application, internet browsing application, electronic mail application, messaging application, and the like. According to an embodiment of the present invention, the application / program 830 comprises a multiple functionality software application for providing word processing functionality, slide presentation functionality, spreadsheet functionality, database functionality and the like.

[0099] The computer 800 in some embodiments can include a variety of sensors 865 for monitoring the environment surrounding and the environment internal to the computer 800. These sensors 865 can include a Global Positioning System (GPS) sensor, a photosensitive sensor, a gyroscope, a magnetometer, thermometer, a proximity sensor, an accelerometer, a microphone, biometric sensor, barometer, humidity sensor, radiation sensor, or any other suitable sensor.

[0100] The following publications are each hereby incorporated herein by reference in their entirety:

[0101] US Patent #11,337,604, Issued: 05 / 24 / 2022, entitled "In vivo object identification, counting, and imaging based on light backscattered from a plane behind the object".

[0102] US Patent #11,185,222, Issued: 11 / 30 / 2021, entitled "Label-free contrast enhancement for translucent cell imaging by purposefully displacing the detector".

[0103] US Patent #10,803,601, Issued: 10 / 13 / 2020, entitled "Rapid assessment and visual reporting of local particle velocity".

[0104] European Patent #3277155, Issued: 09 / 09 / 2020, entitled "Imaging Modalities Using a Reflective Aperture Array in the Imaging Plane to Dynamically Image and Compare Components of the Diffraction Patterns and Imaging Point-Spread Function".

[0105] US Patent # 10,964,944; Issued: 06 / 30 / 2020; entitled "System and Method for Enhanced Contrast Imaging Based on Detection of Different Portions of a Lateral Point-Spread of Light Pattern".

[0106] US Patent #9,844,320, Issued: 12 / 19 / 2017, entitled "System and Method for Observing an Object in a Blood Vessel".

[0107] The disclosures of each and every patent, patent application, and publication cited herein are hereby incorporated herein by reference in their entirety. While this invention has been disclosed with reference to specific embodiments, it is apparent that other embodiments and variations of this invention may be devised by others skilled in the art without departing from the true spirit and scope of the invention.

Claims

CLAIMSWhat is claimed is:

1. A method for capturing and rendering depth information in an eye, comprising: illuminating via a scanning-based instrument an eye structure from a first off-center position; illuminating via the scanning-based instrument the eye structure from at least one second off-center position different from the first; capturing an image of the illuminations in a multiple perspective fashion; and rendering depth information based on the captured image of the illuminations.

2. The method of claim 1, wherein the scanning-based instrument comprises an adaptive optics ophthalmoscope.

3. The method of any of claims 1 to 2, wherein the image of the illuminations is provided to a user in a binocular fashion thus providing for rendering of depth information to the user naturally via stereopsis.

4. The method of any of claims 1 to 3, wherein the image of the illuminations is provided to a computing system.

5. The method of claim 4, wherein the computing system renders the depth information.

6. The method of any of claims 1 to 5, wherein the method is performed in real-time.

7. The method of any of claims 1 to 6, further comprising tracking an object moving into or out of a plane of focus relative to the illuminations.

8. The method of any of claims 1 to 7, wherein the illuminations provide a parallax view of the eye structure.

9. The method of any of claims 1 to 8, further comprising performing multimodal capture comprising at least one of reflectance, phase contrast, absorption, polarization, birefringent, and spectral and fluorescence modalities including one-photon and multiple photon fluorescence.

10. The method of any of claims 1 to 9, wherein the illuminations comprise light of different polarizations.

11. The method of any of claims 1 to 10, wherein the illuminations comprise light of different wavelengths.

12. The method of claim 11, wherein the light of different wavelengths is configured to excite one or more fluorophores.

13. The method of any of claims 1 to 12, wherein the illuminations are applied simultaneously.

14. The method of any of claims 1 to 13, wherein the illuminations are applied sequentially to create frames.

15. The method of claim 14, wherein the frames are interleaved to provide temporal multiplexing.

16. The method of any of claims 1 to 15, wherein one illumination is presented in a forward scan and the other illumination is presented in a backward scan of an ophthalmoscope.

17. The method of any of claims 1 to 16, wherein the depth information is rendered by way of binocular disparity via at least one of a chromatic anaglyph, a polarization goggle, an augmented a virtual display, a lenticular display, and an alternating temporal shutter.

18. A device for capturing and rendering depth information in an eye, comprising: a light source; one or more scanning optical elements configured to direct light from the light source to provide multiple illuminations of an eye structure from at least two off-center positions relative to a pupillary axis; and a capture sensor configured to capture an image of the multiple illuminations in a multiple perspective fashion.

19. The device of claim 18, further comprising means for moving the device relative to the eye.

20. The device of claim 18, further comprising means for moving the eye relative to the device.

21. A system for capturing and rendering depth information in an eye, comprising: an ophthalmoscope configured to perform the method of any of claims 1 to 17.

22. The system of claim 21, further comprising a computing system communicatively connected to the ophthalmoscope, comprising a processor and a non -transitory computer-readable medium with instructions stored thereon, which when executed by the processor, perform steps comprising rendering depth information based on the captured image of the illuminations.

23. The system of any of claims 21 to 22, wherein a full or near full pupil of the eye is utilized for adaptive optics wavefront sensing to provide aberration sensing / correction using one wavelength of light.

24. The system of any of claims 21 to 23, wherein the system is configured to provide at least one of a second wavelength, a polarization multiplexed, and a time multiplexed illumination beam projected simultaneously or sequentially onto a fraction of the pupil to perform decentered axis illumination of the sample. 1

Images