Integrated analysis of multispectral information in ophthalmic applications

Abstract

In some embodiments, an ophthalmic system and computer-implemented method for analyzing multispectral information to generate ophthalmic information is described. In an exemplary ophthalmic system, multispectral information associated with an eye of a patient is captured via an imaging system. A first set of information and a second set of information are extracted from the multispectral information. A visualization of the multispectral information is generated using the first set of information. The first set of information and the second set of information are evaluated using different deep learning models to generate ophthalmic information. Ophthalmic information is sent to the user for diagnostic evaluation.

Description

Background Art

[0001] Multispectral imaging (MSI) and hyperspectral imaging (HSI) techniques are increasingly being used in a variety of applications, including, for example, ophthalmic applications. In the field of ophthalmology, MSI and HSI refer to non-invasive optical imaging modalities that utilize multiple narrow spectral bands of light to image ocular structures (e.g., cornea, retina, meibomian glands, etc.). The differences between MSI and HSI typically revolve around the number of bands and how narrow the bands are. For example, compared to HSI, MSI typically involves a smaller number of wider bands. In contrast, compared to MSI, HSI typically involves a larger number of narrower bands. However, currently, there are limitations to using MSI / HSI data for ophthalmic applications, including, for example, detection of eye diseases. For example, because MSI / HSI data is complex, it is not easy to interpret such data for disease diagnosis. Summary of the invention

[0002] In certain embodiments, an ophthalmic system is provided. The ophthalmic system includes: an imaging system; a memory including executable instructions; and a processor in data communication with the memory. The processor is configured to execute the executable instructions to extract a first set of information from the multispectral information and to extract a second set of information from the multispectral information. The first set of information includes spectral information, and the second set of information includes spatial information. The processor is also configured to execute the executable instructions to: generate a set of ophthalmic information associated with the patient's eye based on evaluating the first set of information using a first deep learning model and evaluating the second set of information using a second deep learning model. The processor is further configured to execute the executable instructions to present an indication of the set of ophthalmic information to a user.

[0003] In certain embodiments, a computer-implemented method is provided. The computer-implemented method includes: receiving multispectral information associated with an eye of a patient. The computer-implemented method also includes: extracting a first set of information from the multispectral information and extracting a second set of information from the multispectral information. The first set of information includes spectral information, and the second set of information includes spatial information. The computer-implemented method also includes: generating a set of ophthalmic information associated with the patient's eye based on evaluating the first set of information using a first deep learning model and evaluating the second set of information using a second deep learning model. The computer-implemented method further includes: presenting an indication of the set of ophthalmic information to a user.

[0004] In certain embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium has computer-executable instructions stored thereon. These computer-executable instructions can be executed by one or more processors to perform operations. The operations include: receiving multispectral information associated with a patient's eye. The operations also include: extracting a first set of information from the multispectral information and extracting a second set of information from the multispectral information. The first set of information includes spectral information, and the second set of information includes spatial information. The operations also include: generating a set of ophthalmic information associated with the patient's eye based on evaluating the first set of information using a first deep learning model and evaluating the second set of information using a second deep learning model. The operations further include: presenting an indication of the set of ophthalmic information to a user. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In order to be able to understand the above-mentioned features of the present disclosure in detail, the present disclosure briefly summarized above can be described in more detail with reference to the embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings only show exemplary embodiments and are therefore not to be considered as limiting the scope thereof, and other equally effective embodiments may be allowed.

[0006] Figure 1 An example ophthalmic system for analyzing multispectral information to generate ophthalmic information in accordance with certain embodiments is presented.

[0007] Figure 2A An example workflow for analyzing multispectral information to generate ophthalmic information in accordance with certain embodiments is presented.

[0008] Figure 2B Another example workflow for analyzing multispectral information to generate ophthalmic information in accordance with certain embodiments is presented.

[0009] Figure 3 Further embodiments of the present invention are described in FIG. 2A to FIG. 2B Some parts of the workflow shown in .

[0010] Figure 4 Further embodiments of the present invention are described in FIG. 2A to FIG. 2B Some parts of the workflow shown in .

[0011] Figure 5 is a flow chart of a method for analyzing multispectral information to generate ophthalmic information according to certain embodiments.

[0012] FIG. 6A to FIG. 6B Example scenarios for analyzing multispectral information to generate ophthalmic information in accordance with certain embodiments are presented.

[0013] Figure 7An example computing system for analyzing multispectral information to generate ophthalmic information in accordance with certain embodiments is presented.

[0014] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation. DETAILED DESCRIPTION

[0015] In an exemplary ophthalmic field such as retinal imaging, MSI / HSI can provide a series of frontal fundus spectral slices, allowing clinicians to simultaneously observe the spatial and spectral characteristics of structures throughout the thickness of the retina. Clinicians can use MSI / HSI data to characterize subtle, deep or overlapping lesions from spectral bands, enabling the diagnosis of eye diseases that may not be diagnosed using human vision alone.

[0016] However, currently, MSI / HSI-based diagnosis remains subjective and very time-consuming, in part due to reliance on manual estimation of MSI / HSI data by clinicians (e.g., eye specialists, such as ophthalmologists). For example, current disease diagnosis methods typically rely on clinicians to extensively evaluate pathological features across spectral bands and compare these features at multiple spatial locations. However, given that MSI / HSI sequences may consist of several spectral bands, interpreting pathology using such a large number of spectral bands is very complex, making disease diagnosis cumbersome, unreliable, and prone to error. Therefore, what is needed are improved systems, devices, and techniques for analyzing MSI / HSI data in ophthalmic applications (e.g., eye disease detection).

[0017] Certain embodiments described herein provide systems, techniques, and devices for analyzing and interpreting MSI / HSI information (or data) in ophthalmic applications (such as disease detection and diagnosis, disease prediction, etc.). More specifically, certain embodiments described below provide a framework for applying image analysis and artificial intelligence (AI) / machine learning (ML) to automatically process and present clinically relevant information from data acquired using MSI / HSI techniques. In certain embodiments, the framework involves separately processing spectral information and spatial information in MSI / HSI information in order to determine meaningful features that can be used for various purposes (such as disease diagnosis, etc.). By separately processing spectral information and spatial information, certain embodiments overcome challenges in interpreting, visualizing, and utilizing MSI / HSI information.

[0018] As used herein, the terms "information" and "data" may be used interchangeably to refer to qualitative observations and / or quantitative data. Additionally, as used herein, the term "multispectral information" may be used to refer to MSI information and / or HSI information. Furthermore, as used herein, hyphenated reference numerals refer to specific instances of elements, while unhyphenated reference numerals refer to elements in general. Thus, for example, device "12-1" refers to an instance of a device class, which may be collectively referred to as device "12", and any one of the device classes may be generally referred to as device "12".

[0019] Figure 1 An example ophthalmic system 100 for analyzing multispectral information (e.g., MSI / HSI information) according to certain embodiments is shown. The ophthalmic system 100 includes an imaging system 150, a computing system 130, a computing system 190, and a display 170. The computing systems 130 and 190 represent various computing systems (or devices), including, for example, laptop computers, mobile computers (e.g., tablet computers or smart phones), server computers, desktop computers, imaging systems, surgical consoles, visualization systems, systems embedded in medical devices, and the like.

[0020] The imaging system 150 represents an imaging system that can capture multispectral information associated with a target (e.g., tissue / structure of the eye 160). The imaging system 150 can include one or more devices / components, such as one or more imaging devices, imaging optics, illumination sources, etc. The imaging device can include a digital camera, microscope, or other imaging device now known or later developed. In an exemplary embodiment, the imaging device(s) include a multispectral imaging device (e.g., a multispectral camera), a hyperspectral imaging device (e.g., a hyperspectral camera), or a combination thereof. The imaging system 150 can employ a variety of different imaging modalities to obtain multispectral information. For example, the imaging system 150 can obtain multispectral information based on reflected light, polarized light, and other light properties.

[0021] The computing system 130 includes a diagnostic tool 135, which is generally configured to perform one or more techniques described herein for analyzing multispectral information in ophthalmic applications. The diagnostic tool 135 includes an analysis component 140, an extraction component 145, and an output component 155, each of which can include hardware components, software components, or a combination thereof. In some embodiments, the diagnostic tool 135 receives input data 110 from an imaging system 150 and generates ophthalmic information 120 based on an evaluation of the input data 110 using the techniques described herein. The input data 110 generally includes multispectral information. In some embodiments, the multispectral information includes MSI data 105. In some embodiments, the multispectral information includes HSI data 115.

[0022] The HSI data 115 typically includes a greater number of narrower spectral bands than the MSI data 105. An exemplary HSI data configuration may include, for example, hundreds of spectral bands in the range of 10 nanometers (nm) to 20 nm. On the other hand, the MSI data 105 typically includes a smaller number of wider spectral bands than the HSI data 115. An exemplary MSI data configuration may include, for example, dozens of spectral bands typically in the range of 20 nm and above.

[0023] In certain embodiments, the diagnostic tool 135 separates the input data 110 into spectral components and spatial components, and analyzes the spectral components and the spatial components separately using one or more of the analysis component 140, the extraction component 145, and the output component 155 to generate the ophthalmic information 120. It should be noted that the analysis component 140, the extraction component 145, and the output component 155 are described in more detail below.

[0024] In some embodiments, the diagnostic tool 135 receives diagnostic criteria 195 (also referred to as diagnostic information) from the computing system 190. The computing system 190 can enable a user (e.g., a clinician, such as an ophthalmologist) to interact with the diagnostic tool 135 when generating the ophthalmic information 120. For example, the diagnostic criteria 195 generally represent input from the user regarding the input data 110. For example, the user can indicate to the diagnostic tool 135 specific spectral bands to focus on, certain diseases for which predictions are to be generated, and so on. In some embodiments, the diagnostic tool 135 generates the ophthalmic information 120 based on the diagnostic criteria 195 and the input data 110.

[0025] The ophthalmic information 120 includes an interpretative image 125 and a diagnostic output 165. The interpretative image 125 may be an interpretative MSI image or an interpretative HSI image. Typically, the interpretative image 125 is a multispectral image that is easier to interpret by a user (e.g., a clinician) than the multispectral image in the input data 110. For example, the interpretative image 125 may include various retinal features present in different spectral bands superimposed on a single frame. Presenting retinal features in this manner (e.g., on a single frame) may help a user interpret, visualize, and / or utilize multispectral information (e.g., retinal features across spectral bands) in a more efficient and / or easier manner when making a disease diagnosis.

[0026] The diagnostic output 165 may include information associated with a prediction of at least one disease of the patient's eye 160 based on the analysis of the input data 110. As shown, the diagnostic output 165 includes or is provided as one or more ophthalmic images 175. Each of the one or more ophthalmic images 175 may have one or more disease indications 180 and one or more severity scores 185 corresponding to the one or more disease indications 180. For example, the ophthalmic image 175 may have an indication of the disease indication 180 overlaid on the ophthalmic image 175 and an indication of the corresponding severity score 185 overlaid on the ophthalmic image 175.

[0027] In some embodiments, the disease indication(s) 180 are segmented regions of the corresponding ophthalmic image 175 that are associated with a particular ocular disease (e.g., a retinal disease). In some embodiments, the severity score 185 includes: (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., a segmented region) within the ophthalmic image 175, and (ii) an indication of the severity of the ocular disease. In an exemplary embodiment, the severity score 185 includes a grade. The grade can be based on a predefined scale (e.g., 0 to 10, 0 to 100), where one end of the scale is associated with the lowest severity and the other end of the scale is associated with the highest severity. In another exemplary embodiment, the severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct). The confidence score can be a percentage. However, it should be noted that these are merely examples of the forms that the severity score(s) 185 may have, and the severity score(s) 185 may have any form / configuration consistent with the functionality described herein.

[0028] In the exemplary diagnostic output 165, the first ophthalmic image 175-1 may have a first disease indication 180-1 of a first retinal disease (e.g., one or more segmented regions in the first ophthalmic image 175-1 indicating the first retinal disease) and a severity score 185-1 of the first retinal disease based on the first disease indication 180-1; the second ophthalmic image 175-2 may have a second disease indication 180-2 of a second retinal disease (e.g., one or more segmented regions in the second ophthalmic image 175-2 indicating the second retinal disease) and a severity score 185-2 of the second retinal disease based on the second disease indication 180-2; and so on.

[0029] The ophthalmic information 120 (including the explanatory images 125 and the diagnostic output 165) is available to a user (e.g., a clinician such as an ophthalmologist), who can use the ophthalmic information 120 to diagnose a disease for a patient. In some embodiments, the ophthalmic information 120 is transmitted to an imaging system 150 for display via the imaging system 150. For example, using augmented reality or other display technology, at least a portion of the ophthalmic information 120 can be presented as an overlay in the field of view (FOV) of the imaging system 150. Additionally or alternatively, in some embodiments, the ophthalmic information 120 is transmitted to a display 170, which can display the ophthalmic information 120 associated with the eye 160 to a user (e.g., an ophthalmologist).

[0030] In the illustrated embodiment, display 170 is separate from imaging system 150 and computing system 130. In some other embodiments, display 170 is integral to imaging system 150 or computing system 130. In still other embodiments, imaging system 150, computing system 130, and computing system 190 are implemented as a single computing system. In some embodiments, display 170 comprises an augmented reality display. In some embodiments, display 170 comprises a virtual reality display. In some embodiments, display 170 comprises a three-dimensional display.

[0031] It should be noted that Figure 1 Reference examples of ophthalmic systems for analyzing multispectral information are shown, and in other embodiments, the ophthalmic systems may have different configurations. Figure 1 The computing system 130, imaging system 150, computing system 190, and display 170 are depicted as separate components, but in some embodiments, the computing system 130, imaging system 150, computing system 190, and display 170 may be part of a single computing system (or device) that analyzes multispectral information, generates ophthalmic information, and presents the ophthalmic information to a user. In general, any number of components (e.g., Figure 1 The ophthalmic system may be implemented using a greater or lesser number of components than those shown).

[0032] Figure 2A An example workflow 200A for analyzing multispectral information to generate ophthalmic information according to certain embodiments is shown. The workflow 200A may be performed by the diagnostic tool 135 .

[0033] In some embodiments, the analysis component 140 includes an analysis tool 240 that is generally configured to separate (or decouple) the multispectral information into spectral components and spatial components. Figure 2AAs shown, for example, the analysis tool 240 receives the input data 110 and extracts the spectral information 205 and the spatial information 210 from the input data 110. The analysis component 140 sends the spectral information 205 and the spatial information 210 to the extraction component 145.

[0034] Spectral information 205 typically represents a raw multi-spectral information sequence (e.g., a raw MSI sequence or a raw HSI sequence). For example, spectral information 205 is typically multi-dimensional and includes information from multiple spectral bands. In contrast, spatial information 210 is typically one-dimensional and may include spatial characteristics of the eye (e.g., the location (including position and / or orientation) of certain features of the eye).

[0035] In some embodiments, the analysis tool 240 performs dimensionality reduction on the input data 110 in order to extract the spatial information 210. An exemplary dimensionality reduction technique (or method) is principal component analysis (PCA). However, it should be noted that PCA is an example and any dimensionality reduction technique consistent with the functionality described herein may be used.

[0036] In some embodiments, the extraction component 145 includes a visualization tool 215, a local feature detection tool 220, and a global feature detection tool 225. The visualization tool 215 is configured to generate an explanatory image 125 based on the spectral information 205. As described above, the explanatory image 125 is a multispectral image that is easier to interpret (by a user) than the multispectral image in the input data 110. For example, the explanatory image 125 can be a single image that includes one or more superimposed features (from one or more spectral bands) from the spectral information 205. The visualization tool 215 can employ deterministic techniques (e.g., weighted averaging) or learning-based techniques (e.g., convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc.) to extract features from the spectral information 205 and superimpose the combined features on a single explanatory image 125. It should be noted that the following description of the explanatory image 125 is not limited to the following description of the explanatory image 125. Figure 3 The visualization tool 215 is described in more detail.

[0037] In some embodiments, the extraction component 145 uses a local feature detection tool 220 and a global feature detection tool 225 to separately process the spectral information 205 and the spatial information 210. For example, the local feature detection tool 220 is generally configured to perform local feature detection based on the spectral information 205, and the global feature detection tool 225 is generally configured to perform global feature detection based on the spatial information 210. The local feature detection tool 220 and the global feature detection tool 225 can use one or more deep learning techniques (or models) (e.g., 2D CNN, 3D CNN, autoencoder, etc.) to perform corresponding local feature detection and global feature detection.

[0038] In some embodiments, the local feature detection tool 220 performs local feature detection by extracting local features 230 from the spectral information 205. The (multiple) local features 230 may correspond to one or more diseased regions of the eye. As used herein, the local features 230 may refer to spectral features extracted from a local region (e.g., a single image pixel) of a spectral image within the spectral information 205. In some embodiments, the global feature detection tool 225 performs global feature detection by extracting global features 235 from the spatial information 210. Similar to the (multiple) local features 230, the global features 235 may also correspond to one or more diseased regions of the eye. As used herein, the global features 235 may refer to spatial range features extracted from a spatial image within the spatial information 210. It should be noted that the following description of the global feature detection tool 225 is not limited to the following description of the global feature detection tool 225. Figure 4 The local feature detection tool 220 and the global feature detection tool 225 are described in more detail.

[0039] The output component 155 receives the local features 230 and the global features 235 and generates a diagnostic output 165. In some embodiments, the output component 155 concatenates the local features 230 and the global features 235 to obtain at least one ophthalmic image 175 having one or more disease indications 180. For example, the at least one ophthalmic image 175 may be a retinal image, and the one or more disease indications 180 may include one or more segmented disease regions within the retinal image. The output component 155 may also provide (as part of the diagnostic output 165) a severity score 185 associated with the disease indication 180. For example, the severity score 185 may indicate a specific retinal disease associated with the segmented disease region and a score indicating the severity of the retinal disease. In addition to or as an alternative to the diagnostic output 165, the output component 155 may provide an interpretive image 125 (received from the visualization tool 215) to the user (e.g., via the imaging system 150 and / or the display 170). As described above, the interpretive image 125 is a single image having one or more features from different spectral bands superimposed on the single image. The interpretative image 125 may enable a user to interpret the multispectral information obtained from the imaging system 150 in a more efficient and / or easier manner when making a disease diagnosis. Figure 4 The output component 155 is described in more detail.

[0040] Figure 2B Another example workflow 200B for analyzing multispectral information to generate ophthalmic information according to some embodiments is shown. The workflow 200B can be performed by the diagnostic tool 135. Figure 2ACompared to the workflow 200A depicted in FIG. 200B, in the workflow 200B, the visualization tool 215 of the extraction component 145 receives the diagnostic criteria 195 as an additional input to the spectral information 205. In some embodiments, a user (e.g., a clinician) can interact with the diagnostic tool 135 via the diagnostic criteria 195. For example, a user can indicate one or more spectral bands of interest, one or more eye features of interest, one or more eye diseases, etc. within the diagnostic criteria 195. The visualization tool 215 can use the diagnostic criteria 195 to generate a customized (or user-defined) explanatory image 125. For example, the visualization tool 215 can filter the amount of spectral features superimposed on the explanatory image 125 according to the diagnostic criteria 195. One benefit of filtering spectral features in this manner is that a single image focusing on a particular spectral feature can be shown to the user, thereby allowing the user to use such features in a more efficient and / or easier manner when making a disease diagnosis.

[0041] Please note that FIG. 2A to FIG. 2B Reference example configurations of workflows 200A and 200B that can be used to analyze multispectral information to generate ophthalmic information are shown, respectively, and workflows 200A and 200B can have other configurations consistent with the functionality described herein. For example, although workflows 200A and 200B are described as being implemented using visualization tools 215, local feature detection tools 220, and global feature detection tools 225, each of workflows 200A and 200B can be implemented using any number of components (e.g., a single component, multiple components, etc.).

[0042] Figure 3 Further embodiments are presented with respect to FIG. 2A to FIG. 2B 1. Certain components of the visualization tool 215 are described. As shown, the visualization tool 215 includes CNNs 3041-K, which are generally configured to process the spectral information 205 to generate an explanatory image 125. In an exemplary embodiment, the spectral information 205 includes different spectral bands 302 (also referred to as spectral frames) of a multispectral image (e.g., an MSI image or an HSI image). In such an embodiment, the visualization tool 215 can extract features from one or more of these spectral bands and overlay these features on a single explanatory image 125.

[0043] Also like Figure 3 As shown, in some embodiments, CNN 304 1-K can receive diagnostic criteria 195 as input. As described above, the user can use diagnostic criteria 195 to configure the type of output to check the diagnosis. For example, explanatory image 125 can be a custom (or user-defined) image with a set of features (from spectral band 302) selected (or configured) by the user.

[0044] It should be noted that although Figure 3 The visualizer 215 is depicted as utilizing one or more CNNs to implement a learning-based technique to process the spectral information 205, but in some embodiments, the visualizer 215 may process the spectral information 205 using other techniques, including, for example, deterministic methods (e.g., weighted averages) or learning-based methods (e.g., RNNs or any other type of deep learning methods).

[0045] Figure 4 Further embodiments are presented with respect to FIG. 2A to FIG. 2B 1. Certain components of the diagnostic tool 135 described. As shown, the spectral information 205 (input into the local feature detection tool 220) includes depth information 404 and multiple spectral features 402 1-2. In some embodiments, the local feature detection tool 220 extracts one or more local features 230 from the spectral information 205 using a deep learning model 410. Each local feature 230 includes a spectral feature 402 from a local area (e.g., a single image pixel).

[0046] The deep learning model 410 can perform an embedding operation to map the spectral information to an internal representation for describing (or labeling) an eye disease. For example, the deep learning model 410 can be trained on a set of spectral features that are labeled with disease labels or functional labels. The deep learning model 410 can predict a classification label associated with the local features 230. In some embodiments, the deep learning model 410 is an RNN. An exemplary RNN is a bidirectional long short-term memory (bidirectional LSTM); however, it should be noted that the prediction tool 430 can use other neural networks consistent with the functionality described herein.

[0047] Also like Figure 4 As shown, the global feature detection tool 225 receives the spatial information 210 and uses the deep learning model 420 to extract the global features 235. In some embodiments, the global features 235 include a set of spectrally encoded spatial information. In some embodiments, the deep learning model 420 is an RNN. However, it should be noted that the deep learning model 420 can be any type of deep learning model.

[0048] The output component 155 generates a diagnostic output 165 based on the local features 230 and the global features 235. In some embodiments, the output component 155 can use the labels corresponding to the local features 230 to establish "words" for describing the spectral data. For example, each "word" can associate a label (corresponding to the local features 230) with a description of a retinal disease symptom (e.g., seeing flashes, blurred vision, reduced central or peripheral vision, sudden loss of vision, changes in color perception) or a specific retinal disease (e.g., retinal tear, retinal detachment, diabetic retinopathy, macular hole, etc.). The output component 155 can then use the global features 235 to combine these "words" into one or more "sentences" that can be used to assess disease patterns or perform early detection. For example, the global features 235 can be used to determine that a group of "words" (that collectively indicate a group of retinal disease symptoms) indicate a disease diagnosis that includes at least one of the following: (i) a specific retinal disease or (ii) the severity of the retinal disease. Additionally or alternatively, in some embodiments, the output component 155 can use a neural network (e.g., a 2D or 3D CNN) or a visual transformer to generate a probability map of disease onset based on the local features 230 and the global features 235. In an exemplary embodiment, the output component 155 can segment choroidal neovascular structures from a retinal image to indicate the presence of age-related macular degeneration.

[0049] Figure 5 is a flow chart of an example method 500 for analyzing multispectral information to generate ophthalmic information in accordance with certain embodiments. The method 500 may be performed by a diagnostic tool (eg, diagnostic tool 135).

[0050] The method 500 enters at block 505 where the diagnostic tool receives multispectral information associated with the patient's eye (e.g., input data 110). In some embodiments, the multispectral information includes MSI data (e.g., MSI data 105). In some embodiments, the multispectral information includes HSI data (e.g., HSI data 115).

[0051] At block 510, the diagnostic tool extracts spectral information (eg, spectral information 205) from the multispectral information and extracts spatial information (eg, spatial information 210) from the multispectral information. In some embodiments, the diagnostic tool performs dimensionality reduction to extract spatial information from the multispectral information.

[0052] At block 515, the diagnostic tool generates a diagnostic output (e.g., diagnostic output 165) associated with the patient's eye based on evaluating the spectral information and the spatial information using different deep learning models. In some embodiments, the diagnostic tool uses a first deep learning model to extract a set of local features (e.g., local features 230) from the spectral information and uses a second deep learning model to extract a set of global features (e.g., global features 235) from the spatial information.

[0053] The diagnostic tool may group the set of local features and the set of global features to obtain a diagnostic output. In some embodiments, the diagnostic output includes one or more ophthalmic images (e.g., (multiple) ophthalmic images 175). Each ophthalmic image may include a disease indication (e.g., disease indication 180). For example, the disease indication may be in the form of one or more segmented regions within the ophthalmic image associated with an ocular disease. The ophthalmic image may also be associated with a severity score (e.g., severity score 185) identifying a particular ocular disease and an indication of the severity of the ocular disease.

[0054] At block 520, the diagnostic tool sends an indication of ophthalmic information (eg, ophthalmic information) to a user (eg, a clinician). The ophthalmic information may include at least a diagnostic output.

[0055] Despite Figure 5 5, but in some embodiments, method 500 includes: generating, by a diagnostic tool, a visualization of multispectral information based on the spectral information. In some embodiments, the visualization includes a single explanatory image (e.g., explanatory image 125) having one or more features from one or more spectral bands of the spectral information. In some embodiments, the diagnostic tool further generates a visualization of the multispectral information based on diagnostic criteria (e.g., diagnostic criteria 195) received from a user (e.g., a clinician) via a computing system (e.g., computing system 190). In some embodiments, the ophthalmic information sent to the user (at box 520) also includes a visualization of the multispectral information.

[0056] FIG. 6A to FIG. 6B An example sequence 600 of analyzing multispectral information to generate ophthalmic information in accordance with certain embodiments is shown. Fig. 6A As shown, the visualization tool 215 evaluates the spectral information 205 including multiple spectral frames 602 to generate the explanatory image 125. As shown, the explanatory image 125 is a single image that includes one or more (superimposed) features 604 from the spectral frames 602.

[0057] like Figure 6BAs shown, the output from the local feature detection tool 220 and the output from the global feature detection tool 225 are used to generate an ophthalmic image 175 with disease indications 180. In the depicted embodiment, the disease indications 180 are segmented areas of retinal disease (e.g., "hard exudates" caused by diabetic macular edema) and have a severity score of "5".

[0058] Figure 7 An example computing system 700 configured to automatically initiate image-guided surgery according to certain embodiments is shown. As shown, the computing system 700 includes, but is not limited to, a processing unit 705, a network interface 715, a memory 720, and a storage device 760, each of which is connected to a bus 717. The computing system 700 may also include an I / O device interface 710 that connects an I / O device 712 (e.g., a keyboard, a display, and a mouse device) to the computing system 700. The computing system 700 is typically under the control of an operating system (not shown). Examples of operating systems include the UNIX operating system, multiple versions of the Microsoft Windows operating system, and distributions of the Linux operating system. (UNIX is a registered trademark of the International Open Standards Organization in the United States and other countries. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.) More generally, any operating system that supports the functionality disclosed herein may be used.

[0059] The processing unit 705 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). The processing unit 705 obtains and executes programming instructions stored in the memory 720 and stored in the storage device 760. The bus 717 is used to transmit programming instructions and application data between the processing unit 705, the I / O device interface 710, the storage device 760, the network interface 715 and the memory 720. It should be noted that the processing unit 705 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, a single GPU, multiple GPUs, a single GPU with multiple processing cores, or any combination thereof. The memory 720 is usually included to represent random access memory. The storage device 760 can be a disk drive or a flash storage device. Although shown as a single unit, the storage device 760 can be a combination of fixed storage devices and / or removable storage devices, such as a fixed disk drive, a removable memory card, an optical storage device, a network attached storage device (NAS) or a storage area network (SAN). Illustratively, the memory 720 includes the diagnostic tool 135, which is discussed in more detail above. Additionally, the storage device 760 includes the input data 110 and the ophthalmic information 120 described above.

[0060] In summary, certain embodiments of the present disclosure provide a framework for applying image analysis and AI / ML techniques to automatically process and present clinically relevant information from data acquired using MSI / HSI techniques. The framework described herein can be used for a variety of ophthalmic applications including disease detection, and can overcome challenges in interpreting, visualizing, and utilizing MSI / HSI information.

[0061] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover any combination of a, b, c, ab, ac, bc, and abc, as well as multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc or any other order of a, b, and c).

[0062] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the claims are not intended to be limited to the embodiments shown herein, but are to be given the full scope consistent with the language of the claims.

[0063] In the claims, unless otherwise specified, reference to a singular element is not intended to mean "one and only one", but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, regardless of whether such disclosure is explicitly recited in the claims, the content disclosed herein is not intended to be committed to the public. According to 35 U.S.C. §112 (f), the elements of any claim will not be interpreted unless the phrase "device for..." is used to expressly recite these elements, or in the case of a method claim, the phrase "step for..." is used to recite these elements. The word "exemplary" is used herein to mean "used as an example, instance, or illustration". Any aspect described as "exemplary" herein is not necessarily interpreted as preferred or advantageous over other aspects.

Claims

1. An ophthalmic system comprising: an imaging system adapted to capture multispectral information associated with an eye of a patient; a memory including executable instructions; as well as a processor in data communication with the memory and configured to execute the executable instructions to perform the following operations: extracting a first set of information from the multispectral information and extracting a second set of information from the multispectral information, wherein the first set of information includes spectral information and the second set of information includes spatial information; generating a set of ophthalmic information associated with the patient's eye based on evaluating the first set of information using a first deep learning model and evaluating the second set of information using a second deep learning model; and An indication of the set of ophthalmic information is presented to a user.

2. The ophthalmic system of claim 1, wherein: The multispectral information includes multispectral imaging (MSI) data.

3. The ophthalmic system of claim 1, wherein: The multispectral information includes hyperspectral imaging (HSI) data.

4. The ophthalmic system of claim 1, wherein: The processor is configured to execute the executable instructions to extract the second set of information by performing a dimensionality reduction operation on the multispectral information.

5. The ophthalmic system of claim 1, wherein: The processor is further configured to execute the executable instructions to perform the following operations: generating a visualization of the multispectral information based on evaluating the first set of information using a third deep learning model; and The visualization is presented to the user.

6. The ophthalmic system of claim 5, wherein: The multispectral information includes a plurality of images, each image being associated with a different spectral band; and The visualization includes a single image including a plurality of features from one or more of the plurality of images.

7. The ophthalmic system of claim 1, wherein: The multispectral information includes a plurality of images, each image being associated with a different spectral band; and The processor is configured to execute the executable instructions to generate the set of ophthalmic information by: (i) extracting a first set of features from one or more regions of one or more images of the plurality of images via the first deep learning model; and (ii) extracting a second set of features from the spatial information via the second deep learning model.

8. The ophthalmic system of claim 7, wherein: The first set of features includes a set of local features; Each of the one or more regions comprises a single pixel from one of the plurality of images; and The second set of features includes a set of global features.

9. The ophthalmic system of claim 7, wherein: The set of ophthalmic information includes a prediction of a condition associated with an eye of the patient.

10. The ophthalmic system of claim 9, wherein: The prediction includes: (i) an ophthalmic image including one or more segmented regions associated with the condition; and (ii) a score indicating a severity of the condition.

Images