Systems and methods for detecting floaters
The system addresses the challenges of manual targeting and real-time tracking in floater treatments by using SLO and OCT imaging with machine learning to detect and treat floaters accurately and safely.
Patent Information
- Application Number
- JP2024532421
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-05-06
- Filing Date
- 2022-11-25
- Publication Date
- 2024-11-15
AI Technical Summary
Current treatments for symptomatic vitreous opacity (SVO), commonly known as floaters, face challenges such as manual targeting risks, limitations in minimum targetable floater size, and the inability to accurately track and treat floaters in real-time due to their independent movement within the eye.
A system utilizing a combination of imaging devices, including scanning laser optometry (SLO) and optical coherence tomography (OCT), along with machine learning algorithms and a treatment laser, to detect, track, and treat floaters in real-time, ensuring safe targeting by determining floater position and movement.
Enables precise and efficient treatment of floaters by accurately tracking and targeting them within the eye, reducing the risk of damaging retinal structures and improving treatment efficacy.
Smart Images

Figure 2024542684000001_ABST
Abstract
Description
[Technical field]
[0001] This application claims priority to Canadian Patent Application No. 3,140,678, filed November 30, 2021, entitled "SYSTEM AND METHOD FOR DETECTION OF FLOATERS," and Canadian Patent Application No. 3,157,811, filed May 6, 2022, entitled "SYSTEM AND METHODS FOR COMBINED REAL-TIME AND NON-REAL-TIME DATA PROCESSING," the entire contents of which are incorporated herein by reference in their entireties.
[0002] The present disclosure relates to systems and methods for detecting and / or treating ophthalmic conditions, and in particular to systems and methods relating to the detection and / or treatment of symptomatic vitreous opacities (SVO), also known as floaters. [Background technology]
[0003] Symptomatic vitreous opacities (SVOs) in a patient's eye, commonly referred to as floaters, impact the patient's vision and / or comfort. Floaters are microscopic fibers that tend to clump together in the vitreous of the eye, casting a shadow on the patient's retina. Current treatments for floaters include removing the vitreous with the floaters and replacing it with a solution. Newer treatments can use lasers to break up the debris in the vitreous. The laser can be aimed at the debris by an ophthalmologist using a targeting laser. Manual targeting procedures run the risk of targeting non-floater elements in the patient's eye. Additionally, manual targeting limits the minimum size of floaters that can be targeted and treated using existing techniques.
[0004] Additional, alternative, and / or improved systems and methods for the detection and / or treatment of one or more ocular conditions are desired. Summary of the Invention
[0005] According to the present disclosure, there is provided a system for use in treating floaters in a patient's eye, the system comprising: a first imaging system for acquiring real-time images of the patient's eye, a laser treatment system for focusing and emitting a treatment laser, and a controller for controlling the first imaging system and the laser treatment system, the controller configured to detect floaters in images acquired by the first imaging system, track a location of the detected floaters in images subsequently acquired by the first imaging system, and focus a treatment laser of the laser treatment system at the tracked location of the detected floaters for subsequently emitting the treatment laser to treat the floaters.
[0006] In a further embodiment of the system, the first imaging system includes a scanning laser ophthalmoscopic imaging system.
[0007] In a further embodiment of the system, the treatment laser includes a femtosecond laser.
[0008] In a further embodiment of the system, detecting floaters is performed using a machine learning algorithm that uses a large scale kernel for object detection.
[0009] In a further embodiment of the system, detecting floater further comprises removing non-floater features of the eye from the image prior to using the machine learning algorithm.
[0010] In a further embodiment of the system, the non-floater features include veins in the eye.
[0011] In a further embodiment of the system, the system further comprises a second imaging system for acquiring real-time images of the patient's eye.
[0012] In a further embodiment of the system, the second imaging system includes an optical coherence tomography (OCT) imaging system.
[0013] In a further embodiment of the system, the locations within the eye that the OCT imaging system images are adjusted based on the tracked locations of the floaters.
[0014] In a further embodiment of the system, an OCT imaging system is used to determine the depth of the floaters.
[0015] In a further embodiment of the system, tracking the position of the detected floater includes stabilizing images subsequently acquired by the first imaging system.
[0016] In a further embodiment of the system, the controller determines one or more of the number of floaters, the surface area of the floaters, the volume of the floaters, the position of the floaters, the opacity of the floaters, the refractive index of the floaters, the speed of movement of the floaters, the direction of movement of the floaters, and the concentration of the floaters.
[0017] In a further embodiment of the system, detecting floaters uses a convolutional neural network (CNN) that takes as input a sequence of multiple (M) image frames acquired by the first imaging system and determines a sequence of M floater detection masks corresponding to the location of floaters in each image frame of the input sequence.
[0018] In a further embodiment of the system, detecting floaters includes applying a CNN to a plurality of input sequences of M image frames, each of which includes a frame of interest, to provide a plurality of floater mask sequences, each of which includes a floater detection mask for the frame of interest, and summing the floater detection masks for the frame of interest from each of the plurality of floater mask sequences.
[0019] In a further embodiment of the system, detecting the floaters further comprises applying a threshold to the sum of the floater detection mask.
[0020] In a further embodiment of the system, detecting floaters and tracking the positions of the detected floaters includes transmitting images acquired by a first imaging system to a remote server to detect floaters in the images, subsequently buffering the images acquired from the first imaging system, receiving the positions of the detected floaters in the images by the remote server, and tracking the positions of the detected floaters in the buffered images.
[0021] In a further embodiment of the system, the controller is further configured to predict a future position of the detected floater.
[0022] In a further embodiment of the system, the system further comprises a visible light imaging system.
[0023] In a further embodiment of the system, the system further comprises a line of sight display.
[0024] In a further embodiment of the system, the gaze display is controlled to allow the patient to move their eyes to affect the movement of the floaters.
[0025] In a further embodiment of the system, the gaze display is controlled to determine the subjective impact of the floaters on the patient's vision.
[0026] In a further embodiment of the system, focusing the treatment laser includes focusing the laser according to a treatment pattern determined for at least a portion of the detected floaters.
[0027] According to the present disclosure, there is provided a method for use in treating floaters, the method including the steps of detecting floaters in acquired images, tracking a location of the detected floaters in the acquired images thereafter, and focusing a treatment laser at the tracked location of the detected floaters for subsequently firing the treatment laser to treat the floaters.
[0028] In a further embodiment of the method, the step of detecting the floaters is performed in a controller of the imaging system.
[0029] In a further embodiment of the method, the step of detecting the floaters is performed on a remote server separate from the controller of the imaging system.
[0030] In a further embodiment of the method, the method further comprises the step of buffering subsequently acquired images.
[0031] In a further embodiment of the method, the method further comprises acquiring a real-time image of the patient's eye with a second imaging system.
[0032] In a further embodiment of the method, the second imaging system includes an optical coherence tomography (OCT) imaging system.
[0033] In a further embodiment of the method, the location within the eye that the OCT imaging system images is adjusted based on the tracked location of the floater.
[0034] In a further embodiment of the method, the method further comprises determining the depth of the floater using an OCT imaging system.
[0035] In a further embodiment of the method, the step of tracking a position of the detected floater includes stabilizing images subsequently acquired by the first imaging system.
[0036] In a further embodiment of the method, stabilizing the image includes tracking retinal motion to determine the stabilized motion.
[0037] In a further embodiment of the method, the controller determines one or more of the number of floaters, the surface area of the floaters, the volume of the floaters, the position of the floaters, the opacity of the floaters, the refractive index of the floaters, the speed of movement of the floaters, the direction of movement of the floaters, and the concentration of the floaters.
[0038] In a further embodiment of the method,
[0039] In a further embodiment of the method, the step of detecting floaters uses a convolutional neural network (CNN) that takes as input a sequence of multiple (M) image frames acquired by the first imaging system and determines a sequence of M floater detection masks corresponding to the location of floaters in each image frame of the input sequence.
[0040] In a further embodiment of the method, detecting the floaters further comprises applying a threshold to the sum of the floater detection mask.
[0041] According to the present disclosure, there is provided a non-transitory computer readable medium having instructions stored thereon which, when executed by a processor of a computing device, configure the computing device to provide a method according to any of the methods described above. [Brief description of the drawings]
[0042] Further features and advantages of the present disclosure will become apparent from the following detailed description considered in conjunction with the accompanying drawings.
[0043] [Figure 1] 1 shows a system for detection and treatment of floaters. [Diagram 2] 1 illustrates an exemplary graphical user interface for use in detecting and treating floaters. [Diagram 3] A method for floater detection and indication is presented. [Figure 4] A method of targeting a laser for use in treating floaters is shown. [Diagram 5] 4 shows a floater detection process. [Figure 6] 4 illustrates a further floater detection process. [Figure 7] 1 shows a distributed system for the treatment of floaters. [Figure 8] 13 illustrates a further method for targeting the laser used to treat floaters. [Figure 9] 1 shows a distributed system for floater detection. [Figure 10A] 1 shows an image of an eye with floaters. [Figure 10B] 8B shows an image of the eye from FIG. 8A with floaters identified. [Figure 11] An SLO image and the corresponding OCT image are shown. [Figure 12] 1 shows the results of a simulation of pressure waves within the eye. [Figure 13] 1 illustrates the use of a laser to affect the movement of the floaters. [Figure 14] 1 shows an optical device having a gaze target display. [Figure 15] 13 shows the use of gaze targets to influence floater movement. [Figure 16] Demonstrates the use of line of sight targets to influence floater movement. [Figure 17A] 1 illustrates an exemplary SVO. [Figure 17B] 17B shows a target volume surrounding the SVO of FIG. 17A. [Figure 18] 1 shows a process for nanoparticle-mediated laser treatment of floaters. [Figure 19] 1 shows an optical system including independent fail-safe hardware. [Figure 20] 1 shows an optical imaging system. [Figure 21]1 shows a process for training a machine learning model to classify floaters. [Figure 22] 1 shows an optical system that uses a trained machine learning model to classify floaters. [Diagram 23] A method of treating a patient with floaters is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0044] Symptomatic vitreous opacities (SVO), commonly referred to as floaters, in a patient's eye can be detected using optical imaging and processing techniques. Detected SVO can be detected in real time and, in some cases, tracked. Detection of SVO is used to evaluate the patient's ocular condition, determine treatment options, and / or treat SVO with a therapeutic laser. Treatment can include ablation or removal or vaporization or liquefaction of SVO or portions thereof by a process of photoionization caused by one or more laser pulses.
[0045] Current imaging and targeting technology does not provide direct feedback to the physician whether the floater is within the safe treatment zone, i.e., too close to the retina or lens. Therefore, a system is needed that can image floaters in the eye and determine whether it is safe to treat them. Furthermore, since floaters move independently of the eye, it is important to deliver 1000 laser pulses quickly to the floater. Shock waves generated by the laser pulses can cause the floater to move, so it is desirable to deliver the pulse quickly before the floater moves. Furthermore, current technology does not allow for real-time 3D imaging of the eye / floater. Although it is possible to image the volume with OCT technology, it takes nearly a second to obtain a volume scan in the best case scenario. Thus, a methodology to image, detect, and track the eye and floater in real time is crucial to always confirm its location, determine whether it is within the safe treatment zone, and finally automatically deliver the laser pulses quickly, accurately, and effectively to remove / reduce the floater.
[0046] The detection and tracking of SVO can be performed in various ways, as further described below, using one or more different imaging devices. For example, a first imaging device, such as a scanning laser ophthalmoscopy (SLO) imaging device, can acquire an image of the eye or a portion of the eye in which the floater is visible. The SLO image may not acquire an image of the actual SVO, but rather an image of the shadow of the SVO on the retina. The image from the first imaging device can provide an XY image from which the location of the floater can be partially determined, but depth information regarding the location of the floater may not be determined by the first imaging device. The XY position of the floater / laser scanning angle can be used to control the imaging position of a second imaging device, such as an optical coherence tomography (OCT) imaging device, which can acquire depth information. With the images from the first and second imaging devices, the 3D location of the floater in the eye can be determined. A combination of multiple imaging devices acquiring images in real time can perform 3D tracking of the floater in real time. The tracking information can be used for various purposes, including, for example, measuring the details of the floater and possibly treating the floater with a laser. Such a laser may be a femtosecond laser with parameters of 1-20 uJ / pulse, a central wavelength of 1030 nm, a repetition rate of 1 KHz-2 MHz, and 100-300 fs / pulse.
[0047] FIG. 1 illustrates a system for the treatment of floaters. The system includes an imaging and therapy device 102 that can be used to image a patient's eye, shown as eye 104. The patient's eye has one or more floaters 106. Although the imaging and therapy device 102 is shown as a single device in FIG. 1, the components may be provided in multiple separate devices. International Patent Application No. PCT / CA2021 / 051451, filed October 15, 2021 and entitled "OPHTHALMOLOGICAL IMAGING AND LASER DELIVERY DEVICE, SYSTEM AND METHODS," which is incorporated herein by reference in its entirety, describes an imaging and therapy device that can be used as the imaging and therapy device 102. The imaging and therapy device 102 includes an SLO imaging device 108 capable of acquiring an XY image 110 of the patient's eye, and an OCT imaging device 112 that acquires a depth image 114 of the patient's eye. The OCT imager 112 can acquire depth "slice" images at specific horizontal locations within the eye. Both imagers 108, 112 can acquire multiple frames of images to provide real-time images or videos of the patient's eye. The imaging components can further include a visual imager 116 that acquires 2D images using a 2D light sensor. The visual imager can use a non-coherent light source. As described in more detail below with reference to FIG. 2, images acquired from the imagers can be used in generating a graphical user interface (GUI).
[0048] The imaging and treatment device 102 may include a treatment laser 118 that can be aimed and fired at a specific location in the patient's eye, such as a floater. The laser may be one of a variety of known treatment lasers, including, for example, a femtosecond laser. Other lasers may be used, including, for example, nanosecond, picosecond, microsecond, millisecond, or continuous wave (cw) lasers. The SLO imager 108, OCT imager 110, and treatment laser 118 may be calibrated such that all of the coordinate systems of the devices are optically aligned or co-registered, such that a location in one of the coordinate systems of the devices is aligned with the same location in the coordinate system of the other device. Optical alignment is achieved by adjusting the optical paths of the different imaging and / or treatment devices to be physically aligned with each other. Co-registration can be achieved by using software techniques that adjust the images or coordinates of the different optical systems so that corresponding locations are in the same location. Optical alignment and / or co-registration can be accomplished in a variety of ways, including those described in PCT Publication WO2122 / 077117, filed October 15, 2012, and entitled "OPTHALMOLOGICAL IMAGING AND LASER DELIVERY DEVICE, SYSTEM, AND METHODS," the entire contents of which are incorporated herein by reference.
[0049] 1, it is understood that each of the imaging devices 108, 112, 116 and the treatment laser 118 includes an optical path and other components, such as a light source, a sensor, etc. The optical paths of the imaging devices and the treatment laser may include at least a portion of the optical path that is common to all devices. For example, the final portion of the optical path, in front of the patient's eye, may be common to all devices.
[0050] The imaging system may include adaptive optics and / or lenses 122 in the optical path of one or more of the imaging and therapy components. As shown in FIG. 1, the adaptive optics / lenses may be positioned such that the optical path of all imaging components passes through the adaptive optics / lenses. However, the adaptive optics / lenses may be positioned such that they are in the optical path of a particular imaging component. The adaptive optics / lenses may be used to adjust the interaction between the laser and the tissue by beam modification. The adaptive optics / lenses may modify one or more characteristics of the laser light, such as the wavefront and / or polarization. The wavefront may be modified, for example, to be Gaussian or non-Gaussian.
[0051] The polarization of one or more of the lasers of the imaging and treatment components may be adjusted to provide various polarizations, such as radial polarization. Radial polarization is useful for providing a small focal spot size of a laser, such as a femtosecond laser used as a treatment laser. Additionally, radial polarization can provide a force to the center of the beam, tending to move or keep debris in a particular location.
[0052] The imaging and therapy components 108, 112, 116 and 118, and adaptive optics / lens 122 are controllable by a controller 124 configured to provide various functions including patient head position / motion detection and tracking, eye position / motion detection and tracking, retina tracking, floater detection function 126, floater tracking function 128, floater path prediction function 130, and floater therapy function 132. The floater detection, path prediction, and floater tracking functions may be provided by the same or similar functions, or may be provided by separate functions.
[0053] For example, the floater detection function 126 can detect floaters in the SLO images using image processing techniques. Detecting floaters, and more specifically, floater shadows, is difficult with current techniques. Current object detection techniques perform well when detecting objects with relatively sharp edges. Machine learning based object detection techniques can use kernels with relatively small kernel sizes, such as 3×3 or 4×4, for feature extraction / detection. Floaters in the acquired SLO images are shadows of actual floaters and typically do not contain sharp edges. To improve floater detection, object detection can be modified to use relatively large kernel sizes, such as 8×8, 16×16, 32×32, or more.
[0054] Additionally, detection of floaters can be further complicated by other features in the image. For example, features such as veins in the eye can make floaters difficult to detect. It is possible to identify non-floater features in an image and remove or mask those features from the image before attempting to detect floaters. Non-floater features can be detected using a variety of image processing techniques, including machine learning image classification techniques and / or object detection techniques.
[0055] It will be appreciated that there are a variety of different patient movements that can complicate floater detection. For example, floaters move through the vitreous of the patient's eye and move through the patient's head. Current systems can isolate the movements, for example, by tracking retinal movement, possibly through software processing, to stabilize the image of the patient's eye. Stabilizing the acquired image can make floaters easier to identify. For example, in an unstabilized video of a patient's eye, the video of the eye, including stationary structures such as veins, macula, etc., appears to move, making it difficult to identify moving floaters. If the video of the eye is stabilized such that stationary structures remain stationary, then identifying moving floaters may be easier.
[0056] The floater path prediction function 130 can predict the future position of one or more floaters. For example, floater tracking is used to predict the position of the floater in future frames using the predicted position. The predicted position is used to speed up floater detection / tracking and treatment. The velocity and future position of the floater can be predicted using a technique that combines the fluid dynamics of the floater's movement in the vitreous with machine learning prediction. Since eye movement affects the movement of the floater, the input to the machine learning predictor can include eye movement measured by imaging techniques such as SLO or other techniques for retinal tracking.
[0057] Although not shown in Figure 1, a calibration device may be provided that models the eye and includes one or more floaters within the calibration device. The calibration device is used to calibrate the device 102 and ensure that it is operating properly. The floaters within the calibration device may have a known size and location within the eye that can be imaged and used to calibrate the dimensions of the imaging system.
[0058] The controller 124 may further include a floater tracking function 128. Regardless of the specific details regarding the image processing used to detect floaters, once detected, floaters may be tracked across subsequently acquired images. Tracking may be performed using traditional image processing or tracking techniques such as optical flow. Additionally or alternatively, tracking may use the same or similar functions as object detection. Traditional techniques may be modified to use additional information from prior tracking. For example, tracking of floaters may be used to predict the location of floaters in future frames, and the predicted location used to speed up floater detection / tracking.
[0059] The tracking function 128 can track the XY position / angle scan of the floater required to image the floater throughout the SLO image. The OCT image or images can be used to track the depth or Z position information of the floater. The tracked XY position of the floater is used to control the location imaged by the OCT imager. Because the OCT imager provides an insufficient depth window to image the entire depth of the patient's eye, it may be necessary to acquire multiple OCT images covering various depths to detect the floater's depth. Once detected, the floater's depth can be tracked and predicted. The predicted floater depth position is used to control at least the initial imaging depth of the OCT image to increase the likelihood that the floater will be imaged by the OCT image. Additionally, multiple OCT images of adjacent depth slices can be acquired to obtain depth information of the entire volume of the floater.
[0060] As previously mentioned, the SLO and OCT imagers 108, 112 are used to detect and track one or more floaters in both the XY image plane of the SLO imager and the XZ and / or YZ or depth image planes of the OCT imager. Reference to the XY and XZ image planes is used for illustrative purposes only, and other relative axes and coordinate systems may be used to provide information regarding the physical location of the floaters. The tracked floater locations may be used by the treatment function 132 of the controller to target the treatment laser 118 at an appropriate location to treat the floaters with the laser. Before firing the treatment laser, the treatment function 132 may check the safety of a potential treatment location. For example, if the floater is in front of and close to the retina, it may be determined that the laser cannot be fired because firing the treatment laser poses too great a risk of hitting the retina. Additionally or alternatively, the treatment function may adjust laser parameters based on the safety level of the treatment location. For example, if there are no other features near the treatment location, the power level of the treatment laser, or the duration of firing the laser, may be increased without posing a danger to the patient's eye.
[0061] The imaging, detection, treatment and tracking of floaters can be performed iteratively. That is, the detection process can be performed continuously to detect floaters. Similarly, the tracking process can be performed continuously to track floaters continuously. Alternatively, the detection process can be performed periodically to detect all floaters and initiate tracking for each detected floater. Periodic detection can be used to update tracking and / or detect new floaters. If detection is performed periodically, it can be performed in a floater process that divides floaters into smaller floaters.
[0062] Detection and tracking of SVO, or more specifically the shadow of SVO on the retina, using a first imaging modality that acquires XY images of the eye, such as SLO images or fundus images, can be used to control the scan path of a second imaging modality that acquires depth or Z axis information, such as an OCT modality. The quality of the imaging depends on the use of the images. For example, if the acquired images are only used for imaging the eye and can then characterize one or more conditions of the patient, the imaging does not need to be done in real time. On the other hand, if the imaging is performed as part of a treatment process, the imaging needs to be completed in real time or near real time.
[0063] In imaging-only mode, an OCT volume scan can be acquired that is slightly larger than the shadow, taking into account the predicted trajectory of the floaters. Since OCT imaging of a volume takes a finite time, e.g., about 1 second, post-processing of the images can be performed to remove motion artifacts caused by floater motion while acquiring a 3D volume.
[0064] Additionally, we describe the second imaging modality that obtains depth information as an OCT imaging modality, but other techniques can be used. For example, the SLO imaging device can be used to obtain depth information of the floaters and image the SVO itself by sweeping the focus of the SLO device through the depth of the vitreous. This allows the SLO imaging device alone to provide real-time imaging of XY as well as depth information of the SVO. Based on the focal depth of the SLO device when the SVO is obtained and in focus, the depth information is determined. Furthermore, SLO imaging may be performed with a femtosecond therapeutic laser as the light source to perform two-photon or multiphoton imaging of the eye. On the detection side, it is possible to reject the wavelength of the femto source and accept only the harmonics of the source. For example, the femtosecond laser source is a 1030 nm laser source, so the second harmonic would be 515 nm. The optical path of the SLO imaging device includes a bandpass filter for the 515 nm harmonic. The floaters are made of collagen. Collagen behaves as a crystal and generates second harmonics that can be acquired and visualized by SLO. The interaction of collagen with a laser source, such as a femtosecond laser, generates various harmonics, and as a result, the second harmonic signal generated by the femtosecond laser source can be relatively strong for floaters compared to the vitreous itself.
[0065] In addition to the second harmonic, collagen in the floaters interacting with the treatment laser, or possibly the imaging laser, may result in a red or blue shift. The resulting red or blue shifted light is filtered and acquired by a suitable sensor. The sensor is an SLO imager, or possibly a floater imager 120, capable of acquiring the red or blue shifted light to image the floaters.
[0066] FIG. 2 illustrates an exemplary graphical user interface for use in detecting and, in some cases, treating floaters. As illustrated in FIG. 2, the GUI 200 can present a variety of information to a user, including, for example, a static SLO image 202 of a patient providing an image of the eye in an XY plane. Although described as an SLO image, the static image 202 can be a fundus image acquired using techniques other than SLO. The static image 202 is obtained from a previously acquired image of the patient's eye. In addition to the static image 202, the GUI 200 can also provide live video or images acquired from various imaging modalities. Two different imaging modalities are depicted in FIG. 2, including an OCT image 204 and a live SLO 206. The OCT image can provide depth information in a Z-axis capture along multiple scan lines, referred to as a B-scan. The static SLO image 202 can include additional information, such as an indication 208 of which scan line or path the OCT scan image 204 was acquired from. Additional information such as the location of ocular features, possibly including veins, unsafe areas, etc., as well as other locations such as treatment locations and unsafe treatment areas, are presented on the static image.
[0067] The live OCT image 204 may display one or more SVOs 210. The SVOs may be highlighted in a variety of ways, including by placing a bounding box around the SVO. Additional information may be overlaid on the OCT image. In addition to the live OCT image, the GUI 200 may also provide a live SLO image, or fundus image 206. The live SLO image may have additional information overlaid on the image. The live SLO image 206 may show the SVO 214, or the shadow of the SVO on the retina. Additional information may be overlaid on the live SLO image 206, including, for example, a bounding box 216 or other highlighting feature of the SVO. Additionally, the path (direction and speed) 218 of the SVO may be highlighted.
[0068] The GUI 200 may provide additional information regarding one or more SVOs, including floater details such as identifier, size, path or trajectory, and other relevant information. Additionally, a 3D representation of the SVO 222 may be provided. The 3D representation may be generated from multiple image frames and / or images, including both SLO and OCT images. The GUI allows for interaction with one or more elements. As an example, one or more of the displayed SVOs may be selected and, in some cases, details including a generated 3D representation of the selected SVO may be presented. Additionally, the GUI 200 may further include one or more elements that allow a user to interact and perform one or more actions, such as initiating treatment 224 to treat one or more of the detected and tracked SVOs.
[0069] Although not depicted in FIG. 2, the GUI can present a variety of different information to the user. For example, a safe zone, i.e., an area, region, or zone within the eye that is safe for treatment with the laser, may be displayed or highlighted on both the SLO and the volumetric or OCT depictions. The safe zone may be provided as a safe zone outline or presented as a colored overlay. Additionally or alternatively, the SVO and / or other features may be highlighted, for example, by covering the SVO and / or features with color when they are within or outside the safe zone. The presented information may include information about the floaters and / or one or more floater characteristics, such as the number of floaters, the surface area of each individual floater, the total surface area of all floaters, the volume of each floater, the total volume of all floaters, the position of the floaters, the opacity of the floaters, the refractive index of the floaters, the speed of movement of the floaters, the direction of movement of the floaters, the concentration of the floaters, etc.
[0070] Additionally, although the GUI depicted in Figure 2 shows a 3D representation of a single floater, it is also possible to provide a 3D visualization of the patient's eye with one or more floaters being tracked. The 3D representation allows the user to better understand whether a particular floater is in a safe position for treatment.
[0071] It will be appreciated that different professionals may prefer different information to be displayed. The GUI allows for customization as to what is displayed and where certain information is displayed. Additionally, the GUI, or other features with which the GUI interacts, may provide functionality that allows the professional to interact with the displayed information. For example, the imaging system may use a high resolution OCT imager, and the GUI may provide functionality that allows the professional to zoom in and out on the live OCT image. The GUI may zoom in on the high resolution OCT image by magnifying a portion of the displayed OCT image. If the OCT image was acquired at high resolution, the portion may be magnified or zoomed in without significantly degrading the displayed image quality. The GUI may provide controls for zooming in and out, such as "+" and "-" buttons. Additionally or alternatively, the zoom function may be controlled by other inputs, such as keyboard keys or combinations of keys and / or mouse buttons. The zoomed display may be useful in providing the professional with a more detailed view of one or more SVOs, which may be desirable in assessing a patient's SVOs and establishing a treatment plan.
[0072] Zooming in / out with an OCT system can also be achieved by controlling the modes of the OCT laser source and detector and digitizer. For example, the resolution can be controlled by changing the wavelength bandwidth of the laser source. However, widening the wavelength bandwidth requires more sampling to maintain the same scan speed. Therefore, the scan speed needs to be controlled to meet the limitations of the digitizer in the system. Furthermore, by widening the OCT imaging window, more samples need to be acquired simultaneously. Therefore, the wavelength bandwidth is reduced or the scan speed is reduced. As an example, an OCT system may come with several pre-set imaging modes. For example, Mode 1 (therapeutic imaging): large imaging window (e.g. 10mm in water), low resolution, and low scan speed of 100khz. Mode 2 (therapeutic mode): short window (e.g. 4mm in water), high resolution, and high speed scan of 240khz. Mode 3 (imaging only): large imaging window (e.g. 10mm in water), high resolution, and high speed of 240khz.
[0073] Zooming in and out may be performed on the SLO by controlling the scan pattern of the SLO galvo resonant scanner.
[0074] FIG. 3 illustrates a method for floater detection and display. Method 300 may be performed in a device such as that shown in FIG. 1 or provided by other devices having various components. Method 300 generates and displays (302) a static SLO image or fundus image of a patient's eye. Generating a static image may include generating an overlay of information on the static image. In addition to generating a static image for display, a real-time SLO image is generated and displayed (304). The real-time SLO image or fundus image is used to determine and track SVO location from the real-time SLO image (306), which is then used to determine OCT scan location / pattern (308). The particular OCT scan may also be based on the characteristics of the SVO in the SLO image. For example, dark areas of the SVO in the SLO are considered more important, and therefore more scans are performed in the dark areas of the SLO image. The OCT scan is displayed (310) and used to obtain depth information of the SVO (312). The GUI displayed in method 300 can display the OCT scan path on a static SLO image and highlight the SVO in both the live OCT scan and the SLO image. The GUI can be used to track and display the SVO. The SVO may be processed with a laser to resolve the SVO. The SVO may be scanned with OCT to generate a 3D volumetric representation.
[0075] FIG. 4 illustrates a method of targeting a laser for use in treating floaters. Method 400 begins with detecting floaters in an image (402). The image is acquired of a plane of the patient's eye, the image being, for example, an SLO image or a regular camera image. Floater detection from an SLO image identifies the location of the floater but does not include depth information. Once the location of the floater is detected (402), its location can be tracked across multiple images (404). Floater tracking (404) provides the location of the floater including depth information. Floater tracking can use images captured with both a first imaging device (i.e., an SLO imager) and a second imaging device (i.e., an OCT imager). Once the location of the floater is tracked, the floater can be treated by targeting a laser to the tracked location (406).
[0076] Floater detection (402) can be performed in a variety of ways. For example, as depicted in FIG. 4, detection begins with detecting, removing, or masking non-floater features in the SLO image (408). Non-floater features are, for example, ocular veins or other natural structures. Non-floater features may be detected using image recognition functions. The image with non-floater features removed or masked is processed using machine learning (ML) object detection to detect floaters (410). Non-floater features can be identified by utilizing fast retinal tracking as described in PCT patent application PCT / CA2022 / 051556, entitled “FAST RETINA TRACKING,” filed October 21, 2022. “FAST RETINA TRACKING” is incorporated herein by reference in its entirety. In “FAST RETINA TRACKING,” the background retinal image is corrected using fast tracker information to make the retina “stationary.” Floaters move independently of the retina and can therefore be identified by detection algorithms.
[0077] ML object detection can use a variety of ML models, including deep learning models, neural networks, and other model architectures. These models are trained using an extensive training process. Object detection may be unsupervised, semi-supervised, or fully supervised. Floater detection may be based on existing ML object detection processes that typically rely on relatively small kernels for feature detection / identification. ML object detection may be modified to use relatively large kernel sizes, for example 16x16, 32x32, or even larger. Increasing the kernel size improves detection of floaters or floater shadows that do not have clearly defined edges in the image.
[0078] Once the initial position of the floater is detected in the SLO image, its position is tracked across multiple frames of the SLO image. In addition to tracking the position of the floater in the SLO image, tracking may also be performed on the OCT image to track the depth of the floater. Tracking can be done in a variety of ways. As depicted, tracking begins with stabilizing the SLO image frames (412). Stabilization is done by registering stationary features in the eye across various frames. Additionally or alternatively, stabilization may be based on eye movement determined by tracking retinal or ocular features. Non-floater features can be tracked using a variety of techniques, including those described in PCT patent application PCT / CA2022 / 051556, entitled "FAST RETINA TRACKING," filed October 21, 2022. "FAST RETINA TRACKING" is incorporated herein by reference in its entirety. Floaters are tracked across different frames of the stabilized image (414) using known techniques such as optical flow. Additionally, tracking can utilize previous tracking information to, for example, predict the likely location of the floater in the current frame to accelerate the tracking process. With the location of the tracked floater in the SLO image frame, the OCT imaging position is adjusted to acquire a depth strip at the location of the floater (416). With the OCT imaging position adjusted, OCT imaging can acquire one or more OCT images that can be processed to determine the depth of the floater (418). The OCT imaging device can only acquire depth slice images over a certain window depth size, which may not cover the entire depth of the patient's eye. Thus, a single OCT image may not capture the floater, and as such, the depth window is adjusted until the floater is captured. The OCT imaging device adjusts the focal depth, allowing the window depth to be changed until the floater is detected in the OCT image. The depth of the floater may be used as the starting depth for subsequent OCT imaging.
[0079] Once the depth and location of the floaters have been tracked, they can be treated (406). The floaters can be treated in a variety of ways, but as shown, the treatment is performed using a laser. The treatment involves aiming (including focusing) a treatment laser at the tracked location / depth of the floaters (420). The safety of firing the laser at the target location is verified (422), and if the treatment location is safe, the laser is fired at the floaters (424) to destroy / ablate / liquefy / vaporize / ablate / photoionize the floaters. Verifying the safety of the target includes determining the proximity of other features of the eye that may be damaged by the laser. If a feature is within the path of the laser or within a threshold distance from the path of the laser, the location is determined to be unsafe for treatment. As will be appreciated, the floaters are moving within the eye, so tracking continues until the floaters are determined to be in a "safe" location for treatment. Validating the safety of the treatment location can take into account the treatment location relative to other features of the eye, as well as other factors such as the power and duration of the treatment laser. Additionally, dynamic safety zones must also be considered: the site within the eye can withstand laser pulses spaced apart by a minimum application time and / or can withstand a maximum number of laser pulses. Laser pulses directed toward certain areas within the eye will displace the floaters, and if the floaters return to those areas, those areas will be considered permanently or temporarily unsafe.
[0080] FIG. 5 illustrates the floater detection process. The process is implemented by hardware such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), or software running on hardware such as an FPGA, ASIC, processor, microprocessor, Graphics Processing Unit (GPU), or Digital Signal Processor (DSP). The process 500 uses a convolutional neural network (CNN) 502 to process a sequence of SLO images 504. The CNN 502 outputs a sequence of masks 506 that provide the detected locations of floaters. The system is trained in a fully supervised manner, where the training ground truth consists of manually annotated SLO images where floaters are annotated. Floaters in captured images are generally out of focus, more so the closer they are to the front of the eye, and the edges are very blurred, typically resulting in a blurry gradient that provides low contrast. Traditional image tracking and object detection typically rely on either (i) landmarks, which are areas of high contrast that are tracked over time, or (ii) edges. In floater detection, the region of interest has very low contrast and no clear edges compared to other features of SLO such as the optic disc, so traditional image tracking processes tend to fail when detecting / tracking floaters.
[0081] The detection process 500 uses a convolutional neural network 502 in a configuration similar to U-Net. Rather than using individual color channels of an image, such as RGB, as input, the input to the CNN 502 contains M channels of images with resolution W1×H1, where M is the number of frames in the SLO sequence. The input can therefore be considered as a sequence of frames of one channel each imaged by the SLO imager. The output includes a segmentation mask 506 indicating the location of the floaters. The output mask has M channels. The resolution of each channel is W2×H2, which does not need to be the same as the input resolution W1×H1.
[0082] The CNN model is trained on a collection of SLO image sequences / videos with labelled floaters. The kernels of the convolutional layers may have larger sizes than typically found in CNNs, such as 8x8, 16x16, 32x32, etc., to accommodate the detection of large feature sizes typical of floaters.
[0083] FIG. 6 illustrates a further floater detection process. To increase the accuracy of the floater detection process 500 described above and to have an adjustable "sensitivity" metric, the process 600 uses multiple image sequences 602 to identify floaters within a single frame. For example, to detect floaters for a sequence length of frame N=20, M=6, floater detection on frame 20 is performed on a sequence of frames including frames 17-22, frame sequence 18-23, etc. Each of these sequences uses a CNN 604 to generate a floater mask prediction for frame 20. This may be the same as previously described in FIG. 5. By predicting over some or all of the frame sequences including frame 20, multiple prediction mask sequences 606 are obtained. Each sequence includes a mask for a frame of interest, i.e., frame 20. Then, the masks of the frames of interest are summed (608). For example, five sequences of images are used, resulting in five different prediction masks for frame 20. If each mask is composed of values ranging from 0 to 1, the sum of the masks will range from 0 to 5. A sensitivity threshold 610 can then be specified between 0 and 5 to fine-tune performance parameters such as false positives and output a smoothed floater mask 612 for the frame of interest. Alternatively, multiple masks can be combined using methods such as multiplication or neural networks.
[0084] Machine learning based floater detection may be combined with classical tracking methods. After detecting floaters using the ML model as described above, the predicted location is passed on to classical image processing based approaches for object tracking, such as optical flow. The predicted motion in the classical image processing based object tracker is used to limit the search area for subsequent ML based floater detection. Additionally or alternatively, after the classical image processing based object tracker is launched for a detected floater, the ML based detection method is periodically launched to re-estimate the floater's location and ensure continued tracking accuracy.
[0085] FIG. 7 illustrates a distributed system for the treatment of floaters. It should be noted that not all possible components of a distributed system are depicted in FIG. 7. For example, device 702 may include additional components such as optical imaging components, additional sensors, adaptive optics / lenses, or other components. The floater imaging and treatment device described above has been described as having a single controller that detects, tracks, and treats floaters. Image processing techniques may require a large amount of processing to perform quickly enough to make real-time tracking and treatment of floaters possible and practical. System 700 may use a remote server, or other remote processing device, to provide the necessary processing requirements for image processing. Remote processing allows for faster or improved image processing, but the additional communication and possibly processing time makes it difficult to provide real-time detection and tracking of floaters. The system 700 described above utilizes an image buffer to enable detection / tracking. System 700 is similar to the floater imaging and treatment device 102 described above, and such similar elements will not be described in further detail.
[0086] The system 700 can transmit acquired images to a remote server 728 via a communication network 730 for processing. The remote server 728 provides an image detection function 720, which performs floater detection and returns the results to the imaging and therapy device 702. There may be a delay in receiving position information of detected floaters from the remote server, which may render the detected positions unsuitable for use in subsequent tracking in the latest images. To address this delay, the device uses an image buffer that can temporarily store images acquired after sending the images to the remote server for detection. Upon receiving the detection results from the remote server 728, the buffered images are used to track the floaters from the detected positions to the current image frame. The controller 718 can use a tracking function 722 substantially similar to the tracking 122 described above, but the tracking may be performed on the buffered images. The tracking is performed relatively quickly, whereby tracking over the buffered images is "fast forwarded" to the current frame, or performed faster than real time, and tracking continues in real time. Such “fast-forward” processing is described in Canadian Patent Application No. 3,157,811, entitled “SYSTEM AND METHODS FOR COMBINED REAL-TIME AND NON-REAL-TIME DATA PROCESSING,” filed May 6, 2022, the contents of which are incorporated by reference in their entirety herein.
[0087] The imaging and therapy device 702 and / or the remote server 728 can communicate with one or more other servers that can provide third party services that can be integrated with storage for image and patient data 732 and other functionality.
[0088] Although the above describes that the detection is performed on a remote server, a similar buffer and fast-forward tracking can be used even if the detection is not performed on a remote server, i.e., if the performed detection process takes time and it is difficult or impossible to use the detected position in the current image as a starting point for tracking, a similar process of buffering images and then fast-forwarding the tracking of the detected position over the buffered images may be applied.
[0089] FIG. 8 illustrates a further method for aiming a laser for use in treating floaters. Method 800 is used to track floater positions from an initial detection position using a detection process that takes a long time making it difficult to use the detected position as an initial tracking position. Method 800 passes an initial image, such as an SLO image, to a floater detection function (802). The floater detection function may be performed locally or remotely. Once the initial floater position is determined, a newly acquired SLO image frame is buffered (804). The detected floater position is received (806) and then used as an initial position for tracking the floater position across the buffered images (808). Tracking of the floater across the buffered images can be performed relatively quickly, allowing tracking across the buffered images to catch up with the currently acquired image. Such processing can be performed using techniques such as those described in Canadian Patent Application No. 3,157,811, entitled "SYSTEM AND METHODS FOR COMBINED REAL-TIME AND NON-REAL-TIME DATA PROCESSING," filed May 6, 2022.
[0090] FIG. 9 illustrates a system for floater detection. It should be noted that not all possible components of a distributed system are depicted in FIG. 9. For example, the device 902 may include further components such as optical imaging components, additional sensors, adaptive optics / lenses, or other components. The system 900 is similar to that described with reference to FIG. 1 and FIG. 5. Similar features and functions will not be described in detail again. The system 900 may include an imaging and treatment device 902 including a first (i.e., SLO) imaging device 108 and a second (i.e., OCT) imaging device 112. However, unlike the devices of FIG. 1 and FIG. 7, the device 902 may omit the floater treatment laser 116, and similarly, the controller 918 may omit the treatment function 124. The controller may include a local detection function 920a and possibly a local tracking function 922a for detecting and tracking floaters, respectively. The local detection and tracking are in conjunction with or replaced by a remote detection function 920b and a remote tracking function 922b provided by a remote server 928 in communication with the device 902 via a communications network 930. The server is remote from the device, but need not be physically separate from the device 902. The controller may also include an image frame buffer 926 and a fast-forward tracking function 928 for tracking the floater from the detected position across the buffered images in the buffer 926.
[0091] While tracking floaters and using tracked locations to target a treatment laser has been described above, tracked floater information may also be used for other purposes. For example, images and locations of floaters may be processed to identify and / or determine characteristics related to one or more floaters. This information may include, for example, the number of floaters, the surface area of individual floaters, the total surface area of all floaters, the volume of individual floaters, the total volume of all floaters, the location of the floaters, the opacity of the floaters, the refractive index of the floaters, the speed of floater movement, the direction of floater movement, the concentration of the floaters, etc. These characteristics may be used for a variety of purposes including, for example, determining the severity of a patient's floater condition, determining appropriate treatment referral pathways, determining the likelihood of successfully treating floaters with a laser, training machine learning models for treatment and diagnostic purposes, etc.
[0092] FIG. 10A shows an image of an eye having a floater. The captured image is a single frame image obtained from an SLO imager. The image 1000 includes at least one floater along with additional features of the eye such as the retina, veins, etc. FIG. 10B depicts the image of the eye of FIG. 10A with a floater identified. The floater is identified by a bounding box 1002, the location of which is used to control the imaging position of the OCT imager. For example, a depth slice is acquired by the OCT imager between the regions identified by lines 1004a, 1004b.
[0093] FIG. 11 shows an SLO image and a corresponding OCT image. The SLO image 1102 provides an XY image of the eye or a portion of the eye. Floaters or shadows of floaters are detected in the image and a scan path for OCT imaging to image the floaters is determined. The scan path is depicted as a diagonal line in a bounding box in FIG. 11. The scan path determined from the SLO image is used to control the scan position of the OCT imager. The OCT imager acquires depth information along the scan path. The OCT imager may acquire depth images 1102, sometimes called b-scan images, along the scan path. The OCT imager may not acquire the entire depth of the eye or vitreous at once. For example, the depth captured in the OCT image 1302 is 2 mm. The 2 mm depth in the eye that the OCT imager acquires is adjusted by adjusting the focus of the imager. The floaters are detected in the OCT image as shown in FIG. 11. If no floaters are detected in the OCT image, the imaging depth can be adjusted to image the eye at various depths along the scan path. If floaters are detected in both the SLO image 1102 and the OCT image 1104, the three-dimensional location of the floaters can be determined. In addition to the floaters, the images 1102, 1104 may be used to determine the location of structures or other features of the eye. For example, veins are clearly visible in the SLO image. Additionally, the surface of the retina is seen at the bottom of the OCT image. Detected or identified structures in the eye, such as the retina, may be used to set or define safety requirements for a treatment or imaging laser. For example, it may not be desirable to focus the laser within 1 mm of the retina. While FIG. 11 illustrates safe or unsafe regions based on proximity to the surface of the retina, more complex safety regions may be defined. For example, the laser may be focused within a certain distance of less than 1 mm from the retina, provided it is below a certain power level or duration, but the laser cannot be focused within a more stringent threshold, regardless of power or duration.
[0094] Thus, a system and method for the detection, tracking, and possible treatment of SVOs has been described. By tracking SVOs in real time, the SVOs can be targeted with a treatment laser and the size of the SVO can be reduced. Real-time tracking, targeting, and treatment of SVOs can be complicated by SVO motion. As described further below, various techniques may be employed to control or at least affect the motion of SVOs.
[0095] FIG. 12 illustrates a simulation of pressure waves within the eye. Pressure waves are transmitted into the eye using a bone transducer, for example, placed on the jawbone. Bone conduction uses relatively low frequency waves that pass into the eye and generate pressure waves within the vitreous humor of the eye. Additionally or alternatively, pressure waves within the eye may be provided by a transducer directly or indirectly coupled to the eye. For example, an ultrasound transducer is coupled to the cornea of the patient's eye. The transducer coupled to the eye may provide higher frequency sound waves to the patient's eye. As shown in FIG. 12, the pressure waves may provide a high pressure region 1202 and a low pressure region 1204 within the patient's eye 1206. The high pressure region tends to push the SVO towards the low pressure region within the eye. Although the SVO may still move within the vitreous humor, the movement of the SVO may be reduced as the SVO tends to move to the low pressure region. This improves the ability to treat the SVO by increasing the time the SVO remains within an area that may be safely targeted by a treatment laser.
[0096] As previously mentioned, pressure waves within the vitreous humor are used to affect the movement of the SVO. Although pressure waves are useful for affecting the movement of the SVO, additional techniques may be used in some cases in combination with the use of pressure waves as previously described.
[0097] FIG. 13 illustrates an optical device with a gaze target display. It should be noted that not all possible components of the distributed system are depicted in FIG. 13. For example, the device 1304 may include additional components such as optical imaging components, additional sensors, adaptive optics / lenses, or other components. The device 1304 may be similar to the devices described above. The device 1304 may include one or more different imaging systems, such as an SLO imaging system 1306, an OCT imaging system 1308, and a treatment laser 1310. In addition to the imaging and treatment systems, the device 1304 further includes a gaze target display 1312. The device includes a controller 1314 that is used to control the imaging and treatment systems as well as the gaze display 1312. The controller 1414 provides gaze control functionality 1316 that can control the gaze target display 1312 to control the patient's eye movements. The gaze control can use information from the imaging system to detect and track SVO. This tracking information may be used to predict the path of SVO and to determine eye movements that can disrupt SVO movements.
[0098] The gaze control function 1316 generates a moving target to move the patient's eyes in a manner that at least partially impedes SVO movement, assuming the patient looks at the target. The target location and movement is programmed to move to induce a specific movement of the patient's eyes. This is used to achieve the desired eye movement and is calculated to cause a predictable and desirable movement in the floater of interest. The generated target may then be displayed on the gaze target display 1312, as depicted by the exemplary target display 1318. The target display 1318 is depicted to provide a moving target "X" for the patient to follow. Although multiple targets are depicted on the display 1318, it is intended to depict a single target that moves over time in a determined manner.
[0099] The display 1318 provides a screen that is displayed to the patient while the eye is being imaged and is used for a variety of purposes. The display is used to identify specific floaters that are symptomatic or bothersome to the patient. The display can display a white screen or a grid to make the floaters more visible to the patient. The physician can use the user interface to select the floaters that are being tracked by the system. The floaters may be highlighted in the patient's field of view using the OCT light or by modifying a portion of the screen. Using this technique, the patient can be quickly guided to help the physician select the correct floater. This can be used for both diagnosing and treating floaters. Other techniques, including using the display to present the user with text to read, may be used to identify specific floaters that are symptomatic or bothersome to the patient. Reading speed, or ability to read, can be used as an indicator of the severity of the tracked SVO that is within the patient's field of view of the text being read.
[0100] FIG. 14 illustrates the use of gaze targets to affect floater movement. An eye 1402 is depicted having a vitreous 1404 with an SVO 1406. The SVO 1406 moves within the vitreous humor as depicted by arrow 1408. Although not depicted in FIG. 14, the location of the SVO can be tracked using one or more imaging systems as previously described. The tracked SVO is targeted with a therapeutic laser to destroy the SVO or otherwise reduce its size.
[0101] A gaze target 1410 is displayed to a user along with a target 1412 that is moved within the gaze target display as depicted by arrow 1414. The target 1412 is moved to impede movement of the SVO. As the patient follows the gaze target 1412, the vitreous humor is caused to move in a similar direction as depicted by arrow 1416. The movement of the vitreous humor caused by movement of the patient's eye following the gaze target can at least partially impede movement of the SVO 1406.
[0102] 15 illustrates a method of using gaze targets to influence floater movement. Method 1500 may be performed by various components of the apparatus. It will be appreciated that the method may be performed in part on the optical device and in part by other computing devices in communication with the optical device.
[0103] The method 1500 determines (1502) the motion of the SVO. Determining the motion of the SVO is performed, for example, by detecting and tracking the SVO and determining its motion and possibly predicted future motion based on the tracking. The detection and tracking and subsequent prediction of future motion may be performed by separate components, possibly using classical computational approaches and / or machine learning models trained to perform a specific task. Additionally or alternatively, one or more of the detection, tracking, and future motion prediction may be combined into a single component. For example, a single machine learning model may be trained to identify SVO and predict future motion of SVO based on a series of images of one or more modalities. Once the motion of the SVO is determined, eye motions that at least partially obstruct the motion of the SVO are determined (1504). The determination of the obstructing motion may be performed in various ways. For example, known motions of the eye are correlated with motions of the vitreous humor, which are used to estimate the effect on the SVO motion. Additionally or alternatively, eye movements and their effects on SVO movement are measured across a range of patients and SVOs to generate an eye movement model that can impede SVO movement. Once an eye movement is determined to at least partially impede SVO movement, a gaze target is displayed to the user to achieve the desired eye movement.
[0104] The gaze target display is manipulated to have the patient move their eyes in a predetermined pattern, and the effect of the movement of one or more SVOs is monitored. This information may be used to determine possible eye movements to counteract the SVO movement. Rather than counteracting the SVO movement, the gaze display is used to induce a pattern of head / eye movement by the patient, which will bring the appropriate floaters into the image and treatment field. Furthermore, a sequence of gaze procedures may be provided to allow all, or substantially all, of the floaters to be detected. Furthermore, the particular sequence and the gaze speed and amplitude of gaze change may be used to extract additional measurements / characteristics of the SVO, such as the floater's location in the vitreous (i.e., distance from the retina), movement speed, opacity, density, etc.
[0105] The gaze target display may also be used for other purposes besides influencing the movement of the SVO. For example, the gaze target display may present text for the patient to read. Since the patient's vision is affected by the SVO, which may be tracked at the same time, the ability to read the displayed text correlates to the tracked SVO that is obscuring the patient's vision. That is, the gaze display may be used to gather subjective information from the patient regarding the severity of the SVO and correlate it to a specific SVO. That is, the gaze display may be used to provide a correlation between what the patient sees and what the physician sees. This may help accurately identify floaters, which is currently a challenge for patients.
[0106] Subjective information collected from patients about the severity of SVO, or the impact of SVO on vision, is combined with the various imaging modalities of SVO to train an SVO classifier that classifies SVO images, and potentially both SLO and OCT modalities, based on the severity of SVO.
[0107] FIG. 16 illustrates the use of a laser to affect floater movement. As previously described, the imaging and treatment device may include a treatment laser, for example using a femtosecond laser as the treatment laser. In addition to using the treatment laser to target SVOs and potentially split or reduce their size, the treatment laser may be used to control or at least affect the movement of the SVO.
[0108] An eye 1602 is depicted having a vitreous 1604 with an SVO 1606. The SVO 1606 moves within the vitreous humor as depicted by arrow 1608. Although not depicted in FIG. 16, the location of the SVO can be tracked using one or more imaging systems as previously described. The tracked SVO is targeted by a treatment laser to destroy the SVO or otherwise reduce its size. The motion of the SVO is influenced by targeting the vitreous along the path of motion of the SVO with a treatment laser. The treatment laser can vaporize the vitreous at the targeted location, creating a small gas bubble 1612 that can act as a barrier to slow down the motion of the SVO. The specific location to target is based on a prediction of the motion of the SVO.
[0109] Thus, various techniques have been described, each of which may affect SVO motion to at least some degree. Although described individually, one or more of the techniques may be used in concert with one another. Additionally, although described with respect to a single SVO, each technique, whether used individually or together, may be used with respect to multiple SVOs.
[0110] As mentioned above, one or more SVOs are detected, tracked, and possibly characterized using one or more imaging modalities. The SVOs can be targeted by a treatment laser. The exact pattern used to target the SVO with the laser can vary. For example, a 3D surface of the SVO is generated based at least in part on the imaging data, and the surface of the SVO is targeted by the laser. While such an approach is possible for treating SVOs, it can be difficult to generate the target pattern required to treat SVOs. Further approaches for defining laser scan patterns, parameters, and techniques for efficiently and effectively treating SVOs are described with respect to Figures 17A and 17B.
[0111] FIG 17A shows an exemplary SVO. Individual SVOs can have a wide range of shapes and sizes. The SVO 1700 depicted in FIG 17A has a large mass or structure 1702 connected to a smaller mass or structure 1706 by a sheet-like structure 1704.
[0112] FIG. 17B shows a geometric target volume of the SVO 1700. As depicted in FIG. 17B, the SVO 1700 is segmented into multiple different parts or regions, and a geometric "target volume" is defined based on 3D imaging of the SVO encompassing one or more of the different regions. The individual target volumes may be positioned such that the entire SVO 1700, or the volumes most important to the treatment of the SVO, are within the target volume. Three separate target volumes are depicted surrounding the SVO. The first target volume 1708 surrounds the SVO mass portion 1702, the second target volume 1712 surrounds the sheet-like connecting structure 1704, and the third target volume 1714 surrounds the second mass structure 1706. The SVO 1700 may be a single SVO or a group of SVOs positioned in a well-defined three-dimensional space.
[0113] Target volumes 1708, 1712, 1714 surrounding a portion of the SVO can be used to generate a laser scan pattern depicted as dots 1710, 1714, 1716. The scan pattern can substantially fill the entire geometric target volume. In addition to the scan pattern, the SVO treatment can define one or more laser parameters such as pulse duration, number of pulses, power, etc. The target volume is defined as a containment zone, where the laser scan pattern and the resulting intravitreal air bubbles provide a boundary for the movement of the floaters during treatment and also serve to stop or direct the movement of the SVO for better treatment safety and efficacy, as previously described. Various target volumes can use a variety of scan patterns that can be determined based on the characteristics of the SVO portion, including its structure, shape, size, movement, and location. As depicted, the scan pattern may proceed in a variety of ways, such as from top to bottom as depicted in scan pattern 1710, from bottom to top as depicted in scan pattern 1714, or other patterns, such as a spiral or circular pattern as depicted in scan pattern 1716. Although the scan pattern is depicted to fill the target volume, it is also possible for the scan pattern to target only the surface of the SVO.
[0114] SVO may be characterized using a collective volume of voxels of 3D image data corresponding to specific locations and optical properties. A voxel is a volumetric unit of a 3D image. The 3D image data is constructed by fusing images of various imaging modalities. For example, 3D image data may be provided by fusing an SLO image and an OCT image of the SVO. Voxels of different zones of the SVO may require different laser and scanning parameters. A laser scan pattern prescribed to fill a geometric target volume with laser pulses of optimized energy and distribution, for example, just above the threshold of photodisruption in a particular voxel, can efficiently photoionize the collective target volume containing the SVO into small bubbles, water, and fine fragments below the threshold of patient symptoms.
[0115] By generating a target volume of an SVO, or a group of SVOs, a geometrically well-defined scan volume for laser treatment can be used, rather than the complex and variable characteristics of the particular SVO being targeted. The use of such a scan pattern can simplify treatment and improve outcomes.
[0116] The scan pattern for treating the SVO is determined in coordination with the scan pattern used in imaging the SVO. The scan pattern for the floater may be a simple geometric grid that fills the volume, or it may be more complex. For example, the scan may use a circular scan pattern, a spiral scan pattern, a linear scan pattern, or a combination of these patterns. During treatment, one possible treatment method is to use a scan and treat approach, where the SVO is first scanned using a circular path to image the floater with OCT and determine where the surface of the floater is located on the Z-axis. After the circular scan to determine the surface, the same circular scan may be repeated again, but using a treatment laser instead of OCT and irradiating the surface of the floater with the femto laser. Imaging and treating the SVO using a circular scan path from outside to inside helps to contain the floater. The focus of the treatment laser may be adjusted based on the SVO surface determined from the previous scan. Such imaging and treatment procedures are beneficial because they allow rapid imaging and subsequent treatment of SVOs in a time period short enough to prevent or substantially limit movement of the SVO between imaging and treatment.
[0117] This allows the SVO to be imaged and treated quickly without giving the floater enough time to move between imaging and treatment. It is also possible to switch scan patterns. For example, with a circular scan pattern, the closer the scan gets to the center, the faster it should be scanned. The scan mode or pattern can be selected based on a variety of factors.
[0118] These laser scan patterns and photoionized bubbles can also be used to restrict floater movement, keep floaters in a zone suitable for treatment, break floaters into small treatable fragments, and aggregate small floaters into larger target volumes. Laser bubble patterns are used to benefit imaging and tracking systems by providing reference targets and boundaries to ensure the target SVO remains in the preferred treatment zone. Small floaters may be surrounded by a ring of bubbles for referencing and treatment sequencing. Additionally, gravity can affect SVO movement, and photoionized bubbles may be generated under the SVO to counteract or reduce gravity-induced movement.
[0119] The treatment center of mass, peripheral features, or other characteristics of the SVO target volume are used for tracking during treatment. The laser pulse energy and treatment parameters on the target are modified to match the characteristics of all local voxels within the geometric target volume containing the SVO, so that treatment planning and execution can be optimized. By creating a well-defined geometric shape, or group of shapes, that encompasses the optimal volume within the vitreous that contains the target SVO, a well-defined laser scan pattern can be performed across the target volume to remove all SVOs within the geometric target volume without employing complex scan patterns based on individual SVO surfaces. This scan pattern for treatment may be optimized for time. A pulse focused on a target volume containing only clear vitreous will only cause small air bubbles and be harmless, while a pulse hitting an SVO will remove the material that makes up the floaters. Other pulses and patterns can be used to contain floater movement, to divide SVOs to treatable or sub-symptomatic sizes, or for other reasons.
[0120] Typically, femtosecond scanning laser patterns are performed from posterior to anterior to avoid plasma shielding during treatment that may cause incomplete or obstructed photoionization. A well-defined geometric target volume, taking into account 3D SVO classification parameters, controlled motion, and floater segmentation, and the most effective volumetric laser scanning method can yield the most effective treatment.
[0121] In this way, classification of SVOs based on groupings of voxels that define local energy and parameter requirements for laser photoionization can be aggregated into larger, well-defined target volumes, allowing for rapid and efficient laser scanning of the entire vitreous region containing the SVO. Similarly, specific peripheral voxels and selected voxel groups can be used for floater containment and controlled treatment methods, making the definition, tracking, and treatment of SVOs more controlled and effective.
[0122] Thus, a possible laser treatment of floaters by tracking and then targeting the floaters with a laser has been described. The aforementioned tracking and targeting process may be combined with other techniques to contain or affect the movement of the floaters. Furthermore, other techniques may be combined with the above techniques to improve the efficacy of the treatment and / or the safety of the treatment. For example, one treatment technique attempts to coat gold nanoparticles (AuNPs) with anionic hyaluronic acid (HA). HA has an affinity for vitreous collagen, which can form floaters. The HA-AuNPs are attracted to the floaters, and subsequent irradiation of the HA-AuNPs with a laser pulse of an appropriate wavelength results in a rapid temperature increase of the AuNP surface. The rapid temperature increase causes the evaporation of the surrounding water, forming vapor nanobubbles. The vapor nanobubbles can rapidly expand and collapse within tens to hundreds of nanoseconds, ablating the collagen of the floaters. Although the use of nanoparticles in this way still requires laser treatment of the eye, the laser power required to create the water vapor nanobubbles can be lower than that required to directly ablate the floaters, making the treatment safer since a lower power laser can be used.
[0123] The aforementioned tracking and targeting of floaters may be combined with nanoparticle-mediated ablation. That is, after injecting HA-coated AuNPs, the floaters can be tracked and targeted with a laser. The HA-AuNPs are attracted to the floaters and tracked and targeted with a laser as described above. Combining the two techniques allows for safer or improved floater ablation with lower power laser pulses. Furthermore, rather than using HA to coat the nanobubble-forming nanoparticles, it is also possible to use other compounds such as indocyanine green (ICG), which has an affinity for collagen. The particles can be injected into the patient's eye and have a certain dimension that allows them to be selectively attracted to the floaters or other structures to be targeted. In addition to their affinity for the targeted structures, the particles have a plasmonic resonance at the wavelength of the treatment laser. The particles lower the power required to cause ablation.
[0124] Additionally, while the above describes the use of nanobubble-forming nanoparticles coated with particles having affinity for the collagen of floaters to safely target and treat floaters using the aforementioned floater targeting and treatment, similar processes can also be used to track and target other structures of the eye or within the vitreous.
[0125] 18 illustrates a process for nanoparticle mediated laser treatment of floaters. The floater treatment process can be performed by an imaging and treatment device 1802 as described above. The process assumes that a patient's eye 1804 has been injected with nanoparticles as described above. The patient's eye may include one or more areas that are unsafe for laser treatment. For example, these areas may include the retina 1806 of the eye and the lens 1808 of the eye. These unsafe areas may be damaged by the laser and adversely affect the patient's vision. As described further below, the tracking and treatment functions may be combined with nanoparticle mediated treatment to safely treat floaters even when in close proximity to unsafe treatment locations within the eye, such as the retina 1806 and / or lens 1808.
[0126] As depicted in FIG. 18, multiple floaters 1810a, 1810b, 1810c (collectively floaters 1810) may be placed within the patient's eye. The floaters 1810 are imaged and tracked by the tracking and treatment device 1802 as they move within the patient's eye. Three different floaters 1810a, 1810b, 1810c are depicted. The first floater 1810a is generally placed in a safe treatment area. This is considered an area where the focused laser light will not damage critical tissue or structures. When treating the floater 1810a in a safe location, the treatment laser of the tracking and treatment device 1802 may be safely focused on the floater 1810a, as depicted by focal point 1812a, to cause ablation. The focused laser light has sufficient power to cause ablation of the floater. Because the floater 1810a is in a safe location, a higher power laser can be used to increase ablation without risk of damaging other tissue. The focused laser is scanned over the floater or a geometric volume containing the floater to cause complete or near complete ablation of the floater 1810a.
[0127] The second floater 1810b is depicted as being in an unsafe position, i.e., close to the lens 1808. The risk of accidentally targeting the floater and accidentally focusing a high power laser on the patient's lens, damaging the lens, may be unacceptably high. By binding nanoparticles to the floater 1810, ablation can occur at power levels lower than the fully focused power level of the laser. Such low power levels are believed to be safe for use in unsafe areas of the retina 1806 and / or lens 1808. Although the power of the laser can be reduced to have a lower power level when fully focused, it is possible to focus a higher power laser behind the floater 1810b such that the focal point is behind the floater, as depicted by focal point 1812b. By focusing a high power laser at a location behind the floater 1810b as depicted in Figure 18, the unfocused laser light impinging on the floater 1810b is at a lower power so that it is safe to use in close proximity to the lens 1808. Although the laser light is unfocused, the laser light on the floater is of sufficient power to cause ablation via the nanoparticles of the floater 1810b. Additionally, the unfocused light can provide power over a larger area while still providing the desired power level.
[0128] Floater 1810c, like floater 1810b, is placed in an unsafe location, i.e., close to the retina 1806. In this case, the laser is focused in front of the floater, i.e., on the anterior side of the floater. To bring the power of the laser light on the retina within safe levels, the laser is focused in front / in front of the floater 1810c, as depicted by focused spot 1812c. The defocused laser light on the retina may not be powerful enough to damage the retina, but is still powerful enough to cause nanoparticle-mediated ablation of the floater 1812c.
[0129] When focusing on the anterior or posterior side of the floater 1810b, 1810c, the floater falls entirely within the region of the defocused laser light and the entire floater is ablated at once. Alternatively, only a portion of the floater 1810b, 1810c falls within the region of the defocused laser light and the laser is steered across the floater to treat different portions of the floater. Alternatively, an unfocused laser is aimed at multiple different locations defined in a scan pattern. The scan pattern may be defined as described above or using alternative techniques.
[0130] Above, imaging and treatment systems and devices that can use lasers for treatment and possibly imaging have been described. Above, various methods have been described for improving the safety of laser treatment, including determining whether treatment can be performed in potentially unsafe areas of the eye, and nanoparticle-mediated treatments to reduce the potential power level of the treatment laser. Each of these techniques can provide sufficient safety, but considering the importance of the patient's eye, additional safety measures can be provided. The safety measures can be used individually or in combination.
[0131] FIG. 19 illustrates an optical system including independent fail-safe hardware. The system is depicted as an imaging and treatment system for an eye 1902 having one or more floaters 1904. Although described with reference to floater treatment, it is understood that the system can be used in other imaging and / or treatment systems that use lasers. The system includes various optical hardware 1906 including one or more lasers 1906a, 1906b, 1906c that can be used for imaging and / or treatment purposes. The optical hardware 1906 can be controlled by a controller 1908, which can be provided by one or more processors, controllers, microcontrollers, etc. The controller controls the optical hardware to provide the desired imaging and treatment functions. The system may further include independent fail-safe hardware 1910 that provides a fail-safe mechanism to prevent the lasers of the optical hardware from damaging the patient's eye.
[0132] The independent fail-safe hardware can operate and monitor signals independent of the controller 1908. The signals include, but are not limited to, SLO laser power, SLO scanner, OCT laser power, OCT scanner, surgical laser power, surgical laser firing time, number of pulses, surgical laser treatment location, eye movement, and other parameters useful in determining the safety of the use of one or more lasers. The independent fail-safe hardware can independently decide to stop firing of any laser in the system with a control line if any danger to the patient is present. Dangers detected by the system include, but are not limited to, firing of SLO laser stationary in the eye, high SLO laser power, firing of OCT laser stationary in the eye, high OCT laser power, wrong location treatment, wrong floater target, excessive surgical laser power, and extended surgical laser firing time. The independent safety hardware can be provided by, for example, ASIC, FPGA, analog electronics, watchdog, DSP, redundant processing units.
[0133] FIG. 20 illustrates an optical imaging system. The optical system 2000 is used to image a patient's eye 2002 having one or more floaters 2004. Although described as providing only imaging functionality, it is also possible for such a device to incorporate treatment functionality. The device includes optical imaging hardware 2006 capable of acquiring images of the patient's eye. The device may be a low-cost device providing only a single imaging modality, such as a fundus imaging device or an SLO device, or it may provide multiple imaging modalities, such as a fundus imaging, SLO imaging and / or OCT imaging system. The optical imaging hardware 2006 is controlled by processing hardware 2008, which includes processing functionality for processing image data directly on the device. Depending on the processing capabilities of the processing hardware, the device can implement various imaging and detection functions as described above. The processing hardware may be provided by an FPGA, an ASIC, or may be provided by a processor / microprocessor. This processing can identify the presence, and possibly severity, of floaters in the patient's eye, and whether the floaters may be a good candidate for laser treatment. Additionally or alternatively, the signal processing on the device is relatively basic and the acquired image data can be passed to another computing device for further processing. The additional computing device can be a local computer 2010 or computing device that processes the acquired images to identify the eye condition and present a graphical user interface (GUI) 2012 to the user. Additionally or alternatively, the processing hardware of the device 2008 can communicate with one or more processing devices via the Internet or other network 2014. The remote processing device can include a remote processing computer for processing the image data and possibly generating a GUI for presentation to the user 2016. The computing device can also include computing devices such as phones or tablets. These can process the image data or simply display the GUI.A device having optical imaging and signal processing hardware can be used to provide a low-cost distributed system that can image a patient's eye and provide an indication of the presence of floaters that can be treated by laser.
[0134] As previously mentioned, the optical imaging hardware 2006 may include a single imaging modality, such as a fundus camera or an SLO imager. Whether the image processing is performed by the signal processing hardware 2008 or one of the remote computing devices 2010, 2016, 2018, machine learning models can be used to identify floaters in the acquired images.
[0135] FIG. 21 illustrates a process of training a machine learning model for classifying floaters. The model may be trained using 2D and 3D data to classify SVOs using only 2D data. The model may simulate a 3D volume of floaters based on the 2D information. The characteristics of SVOs may be generated by the model from the 2D information. The learning of the model may be supervised, semi-supervised, or unsupervised. To train the machine learning (ML) model, a patient with SVO 2102 undergoes SLO imaging 2104, or possibly other fundus imaging. The acquired images are used to identify and track one or more SVOs 2106 in real time. For example, volumetric OCT is performed 2108 to extract SVO characteristics such as the depth of SVOs in the eye, a count of the number of SVOs, the size of SVOs, and the opacity of SVOs 2110. The SVO characteristics extracted from the volumetric OCT data are then used as feedback to train an ML-based classification model 2114. The ML classification model is trained using feedback extracted from the volumetric OCT to classify SVO based on SLO imaging or fundus images alone (2116). The training data from volumetric OCT contains more detailed information than the data from SLO. Once trained, the ML classification model is deployed to one or more imaging devices to classify floaters based on SLO images or fundus images alone.
[0136] FIG. 22 shows an optical device that uses trained machine learning models for imaging, quantification, screening and classification of floaters. The device 2200 can provide a low-cost device that acquires images and processes them using a trained ML classification model to classify SVO in a patient's eye. The classification can provide the patient with an indication of whether the patient may be a candidate for further treatment of SVO. The device 2200 can include an SLO imaging system or optical camera 2202 along with elements such as optical paths and lenses required to image the patient's eye. The device can include a display 2204 that can communicate classification information to the patient. The display can be a simple configuration such as a light or LED that indicates whether or not a patient is a good candidate for treatment, or can include more complex display options such as a display that can display text and or graphic information to the user. The device 2200 can further include a communication radio or modem 2206 that allows the device 2200 to communicate with a remote computing device. The communication radio or modem 2206 can be a wired or wireless radio or modem. The device 2200 includes signal processing hardware 2210 that processes images acquired by an SLO or optical camera 2202. The signal processing hardware 2210 can implement a trained ML classification model 2212 as described above.
[0137] Device 2200 may be provided as a low cost device or possibly integrated into other devices with ocular imaging capabilities. The device may be provided as an in-home screening device that can be used to determine whether a patient should see a specialist for further evaluation.
[0138] FIG. 23 illustrates a method for treating a patient with floaters. The method 2300 may be performed using a single imaging and treatment device, or may be performed using multiple devices. The method includes an initial screening of the patient (2302) to determine whether the patient is a good candidate for further treatment. The screening can be performed by the patient himself using a low-cost screening device that provides only the basic imaging necessary to classify the patient as a good candidate. The low-cost screening device may be provided, for example, as a headset or goggles worn by the patient and that acquires 2D images of the eye (2302a), such as an SLO device, fundus camera, slit lamp, or other similar imaging device. The low-cost device may implement a trained ML model to process the acquired images, as described above, or the device may acquire images and transmit the images for further processing and classification by one or more remote devices. The model may use not only the images, but other information, such as qualitative information provided by the patient. Further information may be extracted from the images and provided to the training model. For example, the depth of the SVO may be estimated based on the relative movements of the various SVOs. The classification may identify SVO in the image, or may simply classify the patient as a good candidate for further treatment (2302b). Once screening determines that the patient is a good candidate for further treatment, the patient may be referred to a specialist with the appropriate equipment to further diagnose and / or treat the patient.
[0139] The patient may see a specialist for further diagnosis and treatment planning (2304). The diagnosis and treatment planning may be provided by a device that includes a higher quality imaging device compared to the screening device. The diagnostic device may have multiple imaging modalities including, for example, an SLO imaging device, an OCT imaging device such as a swept-source OCT that can provide a high resolution scan of the eye, and possibly other imaging devices. Although described as a diagnostic device, it can also be used for screening as described above. The diagnosis may be performed by acquiring 2D and 3D image data of the patient (2304a). Imaging of the patient and the SVO may acquire high resolution images of the SVO. The acquired image data may be processed in a non-real-time manner to improve the quality of the images. The processing may include multiple images, possibly from different imaging systems, together to generate 3D information about the SVO.
[0140] 2D and 3D imaging of SVOs allows individual SVOs to be characterized such as their size, opacity, severity, density, etc. The device may include additional imaging modalities or sensors such as an aberrometer that may be used in characterizing individual SVOs. During 2D and 3D diagnostic imaging, the patient may be instructed to move their eyes by asking them to or by adjusting the gaze target and having the patient track the moving gaze target. Eye movements help ensure, or at least increase the probability, that all floaters are moving within the field of view of the imaging device. Additionally, qualitative information from the patient may be collected and correlated with one or more floaters in the patient's vision. Floaters may be tracked so that when the patient provides qualitative feedback such as the severity of the floaters, it may be correlated with one or more SVOs. The patient may be presented with one or more visual pieces of information such as text or graphics to aid in providing qualitative feedback.
[0141] Each SVO is identified as suitable for laser treatment or not based on the characteristics of the SVO. For SVOs determined to be suitable for laser treatment, a dynamic treatment plan is established by treatment planning (2304b). Dynamic treatment planning is done manually by an expert who explicitly specifies all parameters for treating the SVO. Alternatively, the planning may be automated or partially automated by the system determining the treatment parameters for the SVO. The dynamic treatment plan may determine the scan path, containment pattern, and laser parameters for treating the SVO. The laser parameters may take into account floater characteristics such as floater density, opacity, etc. The dynamic treatment plan for a particular SVO may also specify how the SVO treatment should proceed. For example, an SVO shrinks into multiple smaller SVOs as it is treated, and the treatment plan may provide parameters that specify how to automatically treat such SVOs. For example, the treatment plan may specify the maximum size of the reduced SVOs that will be automatically treated. For larger SVOs, a further treatment plan needs to be determined and approved. This additional planning and approval may be done while the patient is undergoing treatment.
[0142] The dynamic programming process segments the floaters, possibly into multiple voxels as described above, to define the contour of the SVO. The segmentation process is performed using a trained ML model. This process can be fully automated or can allow an expert to provide input to adjust the initial segmentation. Once the expert updates the initial segmentation, the updated segmentation is used to retrain the ML model used. Once the SVO is segmented, a target volume is defined that encloses the SVO, and this target volume is used to define a scan pattern as described with reference to FIG. 17B. Alternatively, the scan pattern may be defined based on the surface of the SVO. If the scan pattern is defined based on the surface of the SVO, the scan pattern needs to be dynamically updated once the initial surface is processed in order to determine the contour of the new surface.
[0143] Once generated, the treatment plan is stored in association with the SVO image data. Storage may be provided on the device or stored on one or more remote devices, including cloud-based storage. The treatment plan may be executed in a real-time treatment process (2306). The real-time treatment (2306) may be executed by a treatment device. The treatment device may download or otherwise access a treatment plan previously prepared for the patient. The treatment device may be similar to the diagnostic device, but further includes at least one treatment laser. The treatment device may be used as both a diagnostic device and a screening device. Diagnosis and planning may be performed separately from the treatment, or both may be performed during a visit to a specialist.
[0144] During real-time treatment, 2D and 3D imaging is performed (2306a). The SVO is tracked in the acquired images (2306b). Once tracked, the SVO is matched with the treatment plan (2306c) and the SVO is treated according to the plan. During real-time floater detection, the 2D and / or 3D images of each detected floater are compared with images of known floaters, each having a specific identifier linked to a floater treatment plan. The treatment plan may be superimposed on the live image of the floater displayed on the GUI. A registration step between the treatment plan and the new orientation of the floater is performed. This may include acquiring a real-time 3D image of the floater and comparing it with the 3D image of the floater of the previously stored treatment plan. This comparison can be performed using a 3D registration based on matching 3D keypoints consisting of feature descriptors, followed by computing the appropriate 3D coordinate transformation from the previously stored 3D floater image to the real-time 3D floater image.
[0145] The real-time processing of SVO can employ one or more of the safety techniques described above. For example, various zones are defined within which it can be determined whether treatment is safe. The zones can be dynamically determined and treatment within the zones can be verified prior to performing the laser treatment or pulse. The safe zones can be dynamically adjusted during the treatment process, for example, based on the amount of pulses delivered to the retina. For example, when a certain amount of pulses or power is delivered to the same area of the retina, that area is dynamically identified as an unsafe area.
[0146] New floaters resulting from the decomposition of larger floaters can be detected and receive a unique identifier that is associated with the original SVO to provide a hierarchy or family tree of SVOs. During treatment, a "dynamic" treatment planning mode is applied to these new floaters. As mentioned above, the treatment plan can specify how to treat the new floaters. The specialist can decide whether each new floater should be treated or not and can also select the desired treatment location in the 3D floater image. Treatment of the SVO can continue until it is determined to be sufficiently treated. The treatment plan for the SVO can include one or more parameters that specify when to stop treatment, such as the size of the shadow on the retina or the volume of the floater. For example, the treatment plan can specify that treatment of the SVO will stop when all particles of floater are below a certain threshold, such as less than 1 mm, less than 0.5 mm, or less than 0.1 mm. When there are no SVOs that can be safely treated, real-time treatment is terminated.
[0147] As mentioned above, the image pressing and treatment device can be used in both imaging and treatment modes. During imaging, the acquired image data does not need to be processed in real time, so high quality image processing can be applied. Furthermore, one SVO can be imaged multiple times as it moves through the eye, and the images can be combined into one high quality 3D representation of the SVO. In the imaging process used for diagnosis and planning, the professional can be presented with both still images of the patient's eye and live images acquired by the various image processing devices. The professional can zoom in and out to specific locations within the images. The zoom in and zoom out capabilities are provided by acquiring a high resolution image and then enlarging an area of the high resolution image. During treatment, the SVO may not need to be imaged in as much detail, so a lower resolution imaging with a larger field of view may be used for the OCT imaging system. For example, the scan lines of the OCT imaging device may be spaced farther apart to create an OCT image at a lower resolution. After treatment, the patient may be re-evaluated to determine whether the treatment was successful. This evaluation may use an imaging system to identify residual SVO, an aberrometer to determine whether the aberrations have improved, or a more qualitative evaluation of the patient.
[0148] Those skilled in the art will appreciate that the systems and components shown in Figures 1-23 may include components not shown. For simplicity and clarity of illustration, the elements in the figures are not necessarily to scale and are merely schematic and non-limiting illustrations of the structures of the elements. It will be apparent to those skilled in the art that numerous variations and modifications may be made without departing from the scope of the invention as defined in the claims.
[0149] Although specific components and steps have been described, the individually described components and steps may be combined into fewer components or steps, or the steps may be performed sequentially, non-sequentially, or simultaneously. Additionally, although the above items have been described as occurring in a particular order, one of ordinary skill in the art in view of the teachings herein will understand that the specific order of certain steps relative to other steps may be changed. Similarly, individual components or steps may be provided by multiple components or steps. One of ordinary skill in the art in view of the present teachings will understand that the components and processes described herein may be provided by various combinations of software, firmware, and / or hardware other than the specific implementations described herein by way of example.
[0150] The techniques of the various embodiments can be implemented using software, hardware, and / or a combination of software and hardware. Various embodiments relate to apparatus, e.g., nodes usable in a communication system or a data storage system. Various embodiments also relate to non-transitory machines, e.g., computers, readable media, e.g., ROM, RAM, CD, hard disk, etc., that include machine-readable instructions for controlling a machine, e.g., a processor, to perform one, several, or all of the described methods or steps of the methods.
[0151] Some embodiments relate to a computer program product including a computer-readable medium including code for causing a computer, or multiple computers, to perform various functions, steps, acts, and / or operations, e.g., one or more or all of the steps described above. In some embodiments, the computer program product may, and sometimes does, include different code for each step performed. Thus, the computer program product may, and sometimes does, include code for each individual step of a method, e.g., a method of operating a communications device, e.g., a wireless terminal or node. The code may be in the form of machine, e.g., computer, executable instructions stored in a computer-readable medium, such as a Random Access Memory (RAM), a Read Only Memory (ROM), or other type of storage device. In addition to relating to computer program products, some embodiments relate to a processor configured to perform one or more of the various functions, steps, acts, and / or operations of the method or methods described above. Thus, some embodiments relate to a processor, e.g., a CPU, configured to perform some or all of the steps of the methods described herein. The processor may, for example, be for use in a communications device or other device described herein.
[0152] Numerous additional variations on the methods and apparatus of the various embodiments as described above will be apparent to those of ordinary skill in the art in view of the above description, and such variations are deemed to be within the scope of this specification.
Claims
1. 1. A system for use in treating floaters in an eye of a patient, comprising: a first imaging system for acquiring a real-time image of the patient's eye; a laser treatment system for focusing and emitting a treatment laser; a controller for controlling the first imaging system and the laser treatment system; Equipped with The controller: Detecting floaters in images acquired by the first imaging system; tracking the location of the detected floaters in images subsequently acquired by the first imaging system; A treatment laser of the laser treatment system is focused on the tracked position of the detected floater for subsequently firing the treatment laser to treat the floater. It was configured as follows: system.
2. The system of claim 1 , wherein the first imaging system comprises a scanning laser ophthalmoscopic imaging system.
3. The system of claim 1 , wherein the treatment laser comprises a femtosecond laser.
4. The system of claim 1 , wherein detecting the floaters is performed using a machine learning algorithm that uses a large scale kernel for object detection.
5. The system of claim 4 , wherein detecting the floaters further comprises removing non-floater features of the eye from the image prior to using the machine learning algorithm.
6. The system of claim 5 , wherein the non-floater features include veins in the eye.
7. The system of claim 1 , further comprising a second imaging system for acquiring real-time images of the patient's eye.
8. The system of claim 7 , wherein the second imaging system comprises an optical coherence tomography (OCT) imaging system.
9. The system of claim 8 , wherein the locations within the eye that the OCT imaging system images are adjusted based on the tracked locations of the floaters.
10. The system of claim 9 , wherein the OCT imaging system is used to determine the depth of the floater.
11. The system of claim 1 , wherein tracking a position of the detected floater includes stabilizing images subsequently acquired by the first imaging system.
12. The controller: Number of floaters, Surface area of the floater, The volume of the floater, Floater position, Floater opacity, The refractive index of the floater, Floater movement speed, The direction of the floater's movement, and Floater concentration, The system of claim 1 , further comprising:
13. detecting the floaters using a convolutional neural network (CNN); The CNN takes as input a sequence of multiple (M) image frames acquired by the first imaging system and determines a sequence of M floater detection masks corresponding to the positions of floaters in each image frame of the input sequence. The system of claim 1 .
14. Detecting the floater includes: applying the CNN to a plurality of input sequences of M image frames, each of which includes a frame of interest, to provide a plurality of floater mask sequences, each of which includes a floater detection mask for the frame of interest; summing the floater detection mask for the frame of interest from each of the plurality of floater mask sequences. The system of claim 13.
15. The system of claim 14 , wherein detecting the floaters further comprises applying a threshold to the sum of the floater detection mask.
16. Detecting the floaters and tracking the positions of the detected floaters includes: transmitting the images acquired by the first imaging system to a remote server for detecting the floaters in the images; buffering subsequent acquired images from the first imaging system; receiving a location of the floater detected in the image by the remote server; and tracking the position of the detected floater in the buffered image. The system of claim 1 .
17. The system of claim 1 , wherein the controller is further configured to predict future positions of the detected floaters.
18. The system of claim 1 further comprising a visible light imaging system.
19. The system of claim 1 further comprising a gaze display.
20. The system of claim 18, wherein the gaze display is controlled to cause the patient to move their eyes to affect the movement of the floaters.
21. The system of claim 18, wherein the gaze display is controlled to determine the subjective impact of the floaters on the patient's vision.
22. The system of claim 1 , wherein focusing the treatment laser includes focusing the treatment laser according to a treatment pattern determined for at least a portion of the detected floater.
23. 1. A method for use in treating floaters, comprising: detecting floaters in the acquired images; tracking the location of the detected floaters in subsequently acquired images; and focusing a treatment laser at the tracked location of the detected floater for subsequently firing the treatment laser to treat the floater. method.
24. The method of claim 23 , wherein the step of detecting the floaters is performed in a controller of an imaging system.
25. 24. The method of claim 23, wherein the step of detecting floaters is performed on a remote server separate from a controller of the imaging system.
26. 26. The method of claim 25, further comprising the step of buffering the subsequently acquired images.
27. 27. The method of claim 26, further comprising acquiring a real-time image of the patient's eye with a second imaging system.
28. 30. The method of claim 27, wherein the second imaging system comprises an optical coherence tomography (OCT) imaging system.
29. 30. The method of claim 28, further comprising adjusting the locations within the eye that the OCT imaging system images based on the tracked locations of the floaters.
30. 30. The method of claim 29, further comprising determining a depth of the floater using the OCT imaging system.
31. 24. The method of claim 23, wherein tracking the position of the detected floater includes stabilizing images subsequently acquired by the first imaging system.
32. 32. The method of claim 31, wherein stabilizing the image includes tracking retinal motion to determine stabilized motion.
33. The controller is Number of floaters, Surface area of the floater, The volume of the floater, Floater position, Floater opacity, The refractive index of the floater, Floater movement speed, The direction of the floater's movement, and Floater concentration, The method of claim 23, further comprising determining one or more of:
34. the step of detecting floaters uses a convolutional neural network (CNN); The CNN takes as input a sequence of multiple (M) image frames acquired by a first imaging system, and determines a sequence of M floater detection masks corresponding to the positions of floaters in each image frame of the input sequence.
24. The method of claim 23.
35. The step of detecting the floater includes: applying the CNN to a plurality of input sequences of M image frames, each of which includes a frame of interest, to provide a plurality of floater mask sequences, each of which includes a floater detection mask for the frame of interest; summing the floater detection mask for the frame of interest from each of the plurality of floater mask sequences.
35. The method of claim 34.
36. 36. The method of claim 35, wherein the step of detecting floaters further comprises applying a threshold to the sum of the floater detection mask.
37. A non-transitory computer readable medium storing instructions that, when executed by a processor of a computing device, configure the computing device to provide a method according to any of claims 23 to 36.
