Integrated analysis of multiple spectral information for ophthalmological applications

Abstract

In certain embodiments, an ophthalmic system and computer-implemented method for analyzing a plurality of spectral information to generate ophthalmic information are described. In the exemplary ophthalmic system, a plurality of spectral information associated with a patient's eye is captured via an imaging system. A first set of information and a second set of information are extracted from the plurality of spectral information. A visualization of the plurality of spectral information is generated using the first set of information. The first set of information and the second set of information are evaluated using different deep learning models to generate ophthalmic information. The ophthalmic information is transmitted to a user for diagnostic evaluation.

Description

[Background technology]

[0001] Multispectral imaging (MSI) and hyperspectral imaging (HSI) techniques are increasingly being used for a variety of applications, including, for example, ophthalmology. In the field of ophthalmology, MSI and HSI refer to non-invasive optical imaging modalities that utilize light in multiple narrow spectral bands to image ocular structures (e.g., the cornea, retina, meibomian glands, etc.). The difference between MSI and HSI generally revolves around the number of bands and how narrow they are. For example, MSI generally involves fewer, broader bands compared to HSI. In contrast, HSI generally involves more, narrower bands compared to MSI. However, there are currently limitations associated with using MSI / HSI data for ophthalmology applications, including, for example, ocular disease detection. For example, due to the complexity of MSI / HSI data, it is not easy to interpret such data for disease diagnosis. Summary of the Invention [Means for solving the problem]

[0002] In certain embodiments, an ophthalmic system is provided. The ophthalmic system includes an imaging system, a memory including executable instructions, and a processor in data communication with the memory. The processor is configured to execute the executable instructions to extract a first set of information from the plurality of spectral information and to extract a second set of information from the plurality of spectral information. The first set of information includes spectral information, and the second set of information includes spatial information. The processor is also configured to execute the executable instructions to generate a set of ophthalmic information associated with the patient's eye based on evaluating the first set of information using a first deep learning model and the second set of information using a second deep learning model. The processor is further configured to execute the executable instructions to present a display of the set of ophthalmic information to a user.

[0003] In certain embodiments, a computer-implemented method is provided. The computer-implemented method includes receiving a plurality of spectral information associated with a patient's eye. The computer-implemented method also includes extracting a first set of information from the plurality of spectral information and a second set of information from the plurality of spectral information. The first set of information includes spectral information, and the second set of information includes spatial information. The computer-implemented method also includes generating a set of ophthalmic information associated with the patient's eye based on evaluating the first set of information using a first deep learning model and the second set of information using a second deep learning model. The computer-implemented method further includes presenting a display of the set of ophthalmic information to a user.

[0004] In certain embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium has computer-executable instructions stored thereon. The computer-executable instructions are executable by one or more processors to perform operations. The operations include receiving a plurality of spectral information associated with a patient's eye. The operations also include extracting a first set of information from the plurality of spectral information and a second set of information from the plurality of spectral information. The first set of information includes spectral information, and the second set of information includes spatial information. The operations also include generating a set of ophthalmic information associated with the patient's eye based on evaluating the first set of information using a first deep learning model and the second set of information using a second deep learning model. The operations further include presenting a display of the set of ophthalmic information to a user.

[0005] So that the above-mentioned features of the present disclosure can be understood in detail, a more particular description of the present disclosure briefly summarized above can be had by reference to embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only exemplary embodiments and therefore should not be considered as limiting the scope of the invention, as other equally effective embodiments are possible. [Brief explanation of the drawings]

[0006] [Figure 1] 1 illustrates an exemplary ophthalmic system for analyzing a plurality of spectral information to generate ophthalmic information, according to certain embodiments. [Figure 2A] 1 illustrates an exemplary workflow for analyzing multiple spectral information to generate ophthalmic information, according to certain embodiments. [Figure 2B] 1 illustrates another exemplary workflow for analyzing multiple spectral information to generate ophthalmic information, according to certain embodiments. [Figure 3] 2A-2B further illustrate certain components of the workflow shown in FIGS. 2A-2B, according to certain embodiments. [Figure 4] 2A-2B further illustrate certain components of the workflow shown in FIGS. 2A-2B, according to certain embodiments. [Figure 5] 1 is a flowchart of a method for analyzing a plurality of spectral information to generate ophthalmic information, in accordance with certain embodiments. [Figure 6A] 1 illustrates an exemplary scenario for analyzing multiple spectral information to generate ophthalmic information, according to certain embodiments. [Figure 6B] 1 illustrates an exemplary scenario for analyzing multiple spectral information to generate ophthalmic information, according to certain embodiments. [Figure 7] 1 illustrates an exemplary computing system for analyzing multiple spectral information to generate ophthalmic information, according to certain embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0007] For ease of understanding, where possible, identical elements common to the figures are designated with the same reference numerals. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further reference.

[0008] In an exemplary ophthalmology application, such as retinal imaging, MSI / HSI can provide a series of en face spectral slices of the fundus, allowing clinicians to visualize both the spatial and spectral characteristics of structures throughout the thickness of the retina. Clinicians can use MSI / HSI data to characterize subtle, deep, or overlapping lesions from spectral bands, enabling the diagnosis of eye diseases beyond those possible using human visual acuity alone.

[0009] However, currently, MSI / HSI-based diagnosis remains subjective and significantly time-consuming, due in part to a reliance on manual estimation of MSI / HSI data by clinicians (e.g., ophthalmologists and other eye specialists). For example, current disease diagnosis methods generally rely on clinicians to broadly evaluate pathological features across spectral bands and compare these features across multiple spatial locations. However, given that MSI / HSI sequences can consist of several spectral bands, interpreting pathology using such a large number of spectral bands can be very complex, making disease diagnosis cumbersome, unreliable, and prone to error. Therefore, what is needed are improved systems, devices, and techniques for analyzing MSI / HSI data for ophthalmological applications, such as eye disease detection.

[0010] Certain embodiments described herein provide systems, techniques, and devices for analyzing and interpreting MSI / HSI information (or data) for ophthalmic applications, such as disease detection and diagnosis, disease prediction, and the like. More specifically, certain embodiments described below provide a framework for applying image analysis and artificial intelligence (AI) / machine learning (ML) to automatically process and present clinically relevant information from data acquired using MSI / HSI techniques. In certain embodiments, the framework includes separately processing spectral and spatial information within the MSI / HSI information to determine significant features that can be used for various purposes, such as disease diagnosis. Certain embodiments overcome challenges in interpreting, visualizing, and utilizing MSI / HSI information by separately processing the spectral and spatial information.

[0011] As used herein, the terms "information" and "data" may be used interchangeably to refer to qualitative observations and / or quantitative data. Additionally, as used herein, the term "multiple spectral information" may be used to refer to MSI information and / or HSI information. Furthermore, as used herein, hyphenated forms of reference numbers refer to specific instances of an element, while unhyphenated forms of reference numbers refer to collective elements. Thus, for example, device "12-1" refers to an instance of a device class, any one of which may be collectively referred to as device "12," and any one of which may be generically referred to as device "12."

[0012] 1 illustrates an exemplary ophthalmic system 100 for analyzing multiple spectral information (e.g., MSI / HSI information) in accordance with certain embodiments. Ophthalmic system 100 includes an imaging system 150, a computing system 130, a computing system 190, and a display 170. Computing systems 130 and 190 represent a variety of computing systems (or devices), including, for example, laptop computers, mobile computers (e.g., tablets or smartphones), server computers, desktop computers, imaging systems, surgical consoles, visualization systems, systems integrated into medical devices, etc.

[0013] Imaging system 150 represents an imaging system capable of capturing multiple spectral information associated with a target (e.g., tissue / structure of eye 160). Imaging system 150 may include one or more devices / components, such as one or more imaging devices, imaging optics, illumination sources, etc. The imaging devices may include digital cameras, microscopes, or other imaging devices now known or later developed. In exemplary embodiments, the imaging devices include multispectral imaging devices (e.g., multispectral cameras), hyperspectral imaging devices (e.g., hyperspectral cameras), or combinations thereof. Imaging system 150 may acquire multiple spectral information using a variety of different imaging modalities. For example, imaging system 150 may acquire multiple spectral information based on reflected light, polarization, and other light properties.

[0014] The computing system 130 generally includes a diagnostic tool 135 configured to perform one or more techniques described herein for analyzing a plurality of spectral information for ophthalmic applications. The diagnostic tool 135 includes an analysis component 140, an extraction component 145, and an output component 155, each of which may include hardware components, software components, or a combination thereof. In certain embodiments, the diagnostic tool 135 receives input data 110 from an imaging system 150 and generates ophthalmic information 120 based on evaluation of the input data 110 using the techniques described herein. The input data 110 generally includes a plurality of spectral information. In certain embodiments, the plurality of spectral information includes MSI data 105. In certain embodiments, the plurality of spectral information includes HSI data 115.

[0015] Compared to MSI data 105, HSI data 115 generally includes a larger number of narrower spectral bands. An exemplary HSI data configuration may include, for example, hundreds of spectral bands in the 10-20 nanometer (nm) range. Meanwhile, compared to HSI data 115, MSI data 105 generally includes fewer broader spectral bands. An exemplary MSI data configuration may include, for example, tens of spectral bands in the 20 nm or greater range.

[0016] In certain embodiments, the diagnostic tool 135 separates the input data 110 into spectral and spatial components and separately analyzes the spectral and spatial components using one or more of an analysis component 140, an extraction component 145, and an output component 155 to generate the ophthalmic information 120. Note that the analysis component 140, the extraction component 145, and the output component 155 are described in more detail below.

[0017] In certain embodiments, diagnostic tool 135 receives diagnostic criteria 195 (also referred to as diagnostic information) from computing system 190. Computing system 190 may allow a user (e.g., a clinician such as an ophthalmologist) to interact with diagnostic tool 135 in generating ophthalmic information 120. For example, diagnostic criteria 195 generally represent input from a user regarding input data 110. For example, a user may indicate to diagnostic tool 135 a particular spectral band to focus on, a particular disease for which a prediction should be generated, etc. In certain embodiments, diagnostic tool 135 generates ophthalmic information 120 based on diagnostic criteria 195 and input data 110.

[0018] The ophthalmic information 120 includes an interpreted image 125 and a diagnostic output 165. The interpreted image 125 may be an interpreted MSI image or an interpreted HSI image. Generally, the interpreted image 125 is a multispectral image that is more easily interpretable by a user (e.g., a clinician) compared to the multispectral images in the input data 110. For example, the interpreted image 125 may include various retinal features present in different spectral bands overlaid in a single frame. Presenting the retinal features in such a manner (e.g., on a single frame) may assist a user in more effectively and / or more easily interpreting, visualizing, and / or utilizing multiple spectral information (e.g., retinal features across spectral bands) when making a disease diagnosis.

[0019] The diagnostic output 165 may include information associated with a prediction of at least one disease of the patient's eye 160 based on the analysis of the input data 110. As shown, the diagnostic output 165 includes or is provided as one or more ophthalmic images 175. Each of the one or more ophthalmic images 175 may have one or more disease indicators 180 and one or more severity scores 185 corresponding to the one or more disease indicators 180. For example, the ophthalmic image 175 may have a representation of the disease indicators 180 superimposed on the ophthalmic image 175 and the corresponding severity scores 185 superimposed on the ophthalmic image 175.

[0020] In certain embodiments, the disease indicators 180 are subregions of the respective ophthalmic images 175 associated with a particular ocular disease (e.g., a retinal disease). In certain embodiments, the severity score 185 includes (i) an indication (or prediction) of the ocular disease for the disease indicator 180 (e.g., a subregion) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease. In an exemplary embodiment, the severity score 185 includes a ranking. The ranking may be based on a predefined scale (e.g., 0-10, 0-100), with one end of the scale associated with the least severity and the other end of the scale associated with the most severity. In another exemplary embodiment, the severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct). The confidence score may be a percentage. However, it should be noted that these are merely examples of forms the severity score 185 may have and that the severity score 185 may have any form / configuration consistent with the functionality described herein.

[0021] In an exemplary diagnostic output 165, a first ophthalmic image 175-1 may have a first disease index 180-1 for a first retinal disease (e.g., one or more divided areas within the first ophthalmic image 175-1 that indicate the first retinal disease) and a severity score 185-1 for the first retinal disease based on the first disease index 180-1, a second ophthalmic image 175-2 may have a second disease index 180-2 for a second retinal disease (e.g., one or more divided areas within the second ophthalmic image 175-2 that display the second retinal disease) and a severity score 185-2 for the second retinal disease based on the second disease index 180-2, and so on.

[0022] The ophthalmic information 120 (including the interpreted image 125 and the diagnostic output 165) is made available to a user (e.g., a clinician such as an ophthalmologist) who can use the ophthalmic information 120 to make a disease diagnosis for the patient. In certain embodiments, the ophthalmic information 120 is transmitted to the imaging system 150 for display via the imaging system 150. For example, at least a portion of the ophthalmic information 120 may be presented as an overlay within the field of view (FOV) of the imaging system 150 using augmented reality or other display technology. Additionally or alternatively, in certain embodiments, the ophthalmic information 120 is transmitted to a display 170, which can display the ophthalmic information 120 associated with the eye 160 to a user (e.g., an ophthalmologist).

[0023] In the illustrated embodiment, display 170 is separate from imaging system 150 and computing system 130. In some other embodiments, display 170 is integral with imaging system 150 or computing system 130. In still other embodiments, imaging system 150, computing system 130, and computing system 190 are implemented as a single computing system. In certain embodiments, display 170 includes an augmented reality display. In certain embodiments, display 170 includes a virtual reality display. In certain embodiments, display 170 includes a three-dimensional display.

[0024] It should be noted that while Figure 1 illustrates a reference example of an ophthalmic system for analyzing multiple spectral information, in other embodiments, the ophthalmic system may have a different configuration. For example, while Figure 1 depicts computing system 130, imaging system 150, computing system 190, and display 170 as separate components, in certain embodiments, computing system 130, imaging system 150, computing system 190, and display 170 may be part of a single computing system (or device) that analyzes multiple spectral information, generates ophthalmic information, and presents the ophthalmic information to a user. In general, the ophthalmic system may be implemented using any number of components (e.g., more or fewer components than those shown in Figure 1).

[0025] 2A illustrates an exemplary workflow 200A for analyzing multiple spectral information to generate ophthalmic information, according to certain embodiments. Workflow 200A may be performed by diagnostic tool 135.

[0026] In certain embodiments, the analysis component 140 generally includes an analysis tool 240 configured to split (or separate) the plurality of spectral information into spectral and spatial components. As shown in Figure 2, for example, the analysis tool 240 receives the input data 110 and extracts the spectral information 205 and spatial information 210 from the input data 110. The analysis component 140 sends the spectral information 205 and spatial information 210 to the extraction component 145.

[0027] The spectral information 205 generally represents a sequence of raw spectral information (e.g., a raw MSI sequence or a raw HSI sequence). For example, the spectral information 205 is generally multi-dimensional and includes information from across multiple spectral bands. In contrast, the spatial information 210 is generally one-dimensional and may include spatial characteristics of the eye (e.g., the location (including the location and / or orientation) of particular features of the eye).

[0028] In particular embodiments, the analysis tool 240 performs dimensionality reduction on the input data 110 to extract the spatial information 210. An exemplary dimensionality reduction technique (or method) is principal component analysis (PCA). However, it should be noted that PCA is an example and any dimensionality reduction technique consistent with the functionality described herein can be used.

[0029] In certain embodiments, the extraction component 145 includes a visualization tool 215, a local feature detection tool 220, and a global feature detection tool 225. The visualization tool 215 is configured to generate an interpretation image 125 based on the spectral information 205. As described above, the interpretation image 125 is a multispectral image that is more easily interpretable (by a user) compared to the multispectral image in the input data 110. For example, the interpretation image 125 may be a single image that includes one or more superimposed features (across one or more spectral bands) from the spectral information 205. The visualization tool 215 may use a deterministic approach (e.g., weighted averaging) or a learning-based approach (e.g., a convolutional neural network (CNN), a recurrent neural network (RNN), etc.) to extract features from the spectral information 205 and superimpose the combined features into a single interpretation image 125. Note that the visualization tool 215 is described in more detail below with respect to FIG. 3 .

[0030] In particular embodiments, extraction component 145 processes spectral information 205 and spatial information 210 separately using local feature detection tool 220 and global feature detection tool 225, respectively. For example, local feature detection tool 220 is generally configured to perform local feature detection based on spectral information 205, and global feature detection tool 225 is generally configured to perform global feature detection based on spatial information 210. Local feature detection tool 220 and global feature detection tool 225 may use one or more deep learning techniques (or models) (e.g., 2D CNN, 3D CNN, autoencoder, etc.) to perform their respective local and global feature detection.

[0031] In certain embodiments, the local feature detection tool 220 performs local feature detection by extracting local features 230 from the spectral information 205. The local features 230 may correspond to one or more disease regions of the eye. As used herein, local features 230 may refer to spectral features extracted from a local region (e.g., a single image pixel) of a spectral image in the spectral information 205. In certain embodiments, the global feature detection tool 225 performs global feature detection by extracting global features 235 from the spatial information 210. Like the local features 230, the global features 235 may also correspond to one or more disease regions of the eye. As used herein, global features 235 may refer to spatial range features extracted from a spatial image in the spatial information 210. Note that the local feature detection tool 220 and the global feature detection tool 225 are described in more detail below with respect to FIG. 4.

[0032] The output component 155 receives the local features 230 and the global features 235 to generate the diagnostic output 165. In certain embodiments, the output component 155 concatenates the local features 230 and the global features 235 to obtain at least one ophthalmic image 175 having one or more disease indicators 180. For example, the at least one ophthalmic image 175 may be a retinal image, and the one or more disease indicators 180 may include one or more segmented disease regions within the retinal image. The output component 155 may also provide (as part of the diagnostic output 165) a severity score 185 associated with the disease indicator 180. For example, the severity score 185 may indicate a particular retinal disease associated with the segmented disease region along with a score indicating the severity of the retinal disease. In addition to or instead of the diagnostic output 165, the output component 155 may provide the interpreted image 125 (received from the visualization tool 215) to the user (e.g., via the imaging system 150 and / or the display 170). As described above, interpretation image 125 is a single image having one or more features from different spectral bands overlaid on a single image. Interpretation image 125 may enable a user to more effectively and / or more easily interpret multiple spectral information obtained from imaging system 150 when making a disease diagnosis. Note that output component 155 is described in more detail below with respect to FIG. 4.

[0033] FIG. 2B shows another exemplary workflow 200B for analyzing multiple pieces of spectral information to generate ophthalmic information, according to certain embodiments. Workflow 200B may be executed by diagnostic tool 135. Compared to workflow 200A depicted in FIG. 2A, in workflow 200B, visualization tool 215 of extraction component 145 receives diagnostic criteria 195 as an additional input to spectral information 205. In certain embodiments, a user (e.g., a clinician) may interact with diagnostic tool 135 using diagnostic criteria 195. For example, the user may indicate one or more spectral bands of interest, one or more ocular features of interest, one or more ocular diseases, etc., in diagnostic criteria 195. The visualization tool 215 may use diagnostic criteria 195 to generate a custom (or user-defined) interpretation image 125. For example, visualization tool 215 may filter the amount of spectral features overlaid on interpretation image 125 based on diagnostic criteria 195. One advantage of filtering spectral features in this manner is that the user can be presented with a single image that focuses on specific spectral features, allowing the user to use such features more effectively and / or more easily when making disease diagnoses.

[0034] 2A-2B illustrate example configurations of workflows 200A and 200B, respectively, that can be used to analyze multiple spectral information to generate ophthalmic information, and workflows 200A and 200B may have other configurations consistent with the functionality described herein. For example, although workflows 200A and 200B are described as being implemented using visualization tool 215, local feature detection tool 220, and global feature detection tool 225, each of workflows 200A and 200B may be implemented using any number of components (e.g., a single component, multiple components, etc.).

[0035] 3 further illustrates certain components of the visualization tool 215 described in connection with FIGS. 2A-2B , according to certain embodiments. As illustrated, the visualization tool 215 generally includes CNNs 304 1-K configured to process the spectral information 205 to generate the interpreted image 125. In an exemplary embodiment, the spectral information 205 includes different spectral bands 302 (also referred to as spectral frames) of a multispectral image (e.g., an MSI image or an HSI image). In such an embodiment, the visualization tool 215 can extract features from one or more of the spectral bands and overlay the features onto a single interpreted image 125.

[0036] 3, in particular embodiments, CNNs 304 1-K may receive as input diagnostic criteria 195. As described above, a user may use diagnostic criteria 195 to configure the type of output to examine for diagnosis. For example, interpretation image 125 may be a custom (or user-defined) image having a set of features (from across spectral bands 302) selected (or configured) by the user.

[0037] It should be noted that while FIG. 3 depicts visualization tool 215 implementing a learning-based approach using one or more CNNs to process spectral information 205, in certain embodiments, visualization tool 215 may use other approaches to process spectral information 205, including, for example, deterministic approaches (e.g., weighted averaging) or learning-based methods (e.g., RNNs or any other type of deep learning method).

[0038] 4 further illustrates certain components of the diagnostic tool 135 described with respect to FIGS. 2A-2B, according to certain embodiments. As shown, the spectral information 205 (input to the local feature detection tool 220) includes depth information 404 and multiple spectral features 402 1-2. In certain embodiments, the local feature detection tool 220 uses a deep learning model 410 to extract one or more local features 230 from the spectral information 205. Each local feature 230 includes a spectral feature 402 from a local region (e.g., a single image pixel).

[0039] The deep learning model 410 may perform an embedding operation to map the spectral information to an internal representation that is used to describe (or label) the ocular disease. For example, the deep learning model 410 may be trained with a set of spectral features that have been labeled with a disease or function label. The deep learning model 410 may predict a classification label associated with the local features 230. In particular embodiments, the deep learning model 410 is an RNN. An exemplary RNN is a bidirectional long short-term memory (bidirectional LSTM), although it should be noted that the prediction tool 430 may use other neural networks consistent with the functionality described herein.

[0040] 4, the global feature detection tool 225 receives the spatial information 210 and extracts global features 235 using a deep learning model 420. In particular embodiments, the global features 235 include a set of spectrally encoded spatial information. In particular embodiments, the deep learning model 420 is an RNN. However, it should be noted that the deep learning model 420 can be any type of deep learning model.

[0041] The output component 155 generates a diagnostic output 165 based on the local features 230 and the global features 235. In particular embodiments, the output component 155 may establish “words” to describe the spectral data using labels corresponding to the local features 230. For example, each “word” may associate a label (corresponding to the local features 230) with a description of a retinal disease symptom (e.g., flashes of light, blurred vision, decreased central or peripheral vision, sudden vision loss, changes in color vision) or a specific retinal disease (e.g., retinal tears, retinal detachment, diabetic retinopathy, macular hole, etc.). The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate disease patterns or perform early detection. For example, the global features 235 may be used to determine that a group of “words” (collectively indicative of a set of retinal disease symptoms) is indicative of a disease diagnosis, including at least one of (i) a specific retinal disease or (ii) the severity of the retinal disease. Additionally or alternatively, in certain embodiments, output component 155 can use a neural network (e.g., a 2D or 3D CNN) or a vision transformer to generate a probability map of disease onset based on local features 230 and global features 235. In an exemplary embodiment, output component 155 can segment choroidal neovascularization structures from a retinal image to indicate the presence of age-related macular degeneration.

[0042] 5 is a flowchart of an exemplary method 500 for analyzing a plurality of spectral information to generate ophthalmic information, according to certain embodiments. Method 500 may be performed by a diagnostic tool (e.g., diagnostic tool 135).

[0043] Method 500 begins at block 505, where a diagnostic tool receives a plurality of spectral information (e.g., input data 110) associated with a patient's eye. In certain embodiments, the plurality of spectral information includes MSI data (e.g., MSI data 105). In certain embodiments, the plurality of spectral information includes HSI data (e.g., HSI data 115).

[0044] At block 510, the diagnostic tool extracts spectral information (e.g., spectral information 205) from the plurality of spectral information and spatial information (e.g., spatial information 210) from the plurality of spectral information. In certain embodiments, the diagnostic tool performs dimensionality reduction to extract the spatial information from the plurality of spectral information.

[0045] At block 515, the diagnostic tool generates a diagnostic output (e.g., diagnostic output 165) associated with the patient's eye based on evaluating the spectral information and the spatial information using different deep learning models. In certain embodiments, the diagnostic tool uses a first deep learning model to extract a set of local features (e.g., local features 230) from the spectral information and a second deep learning model to extract a set of global features (e.g., global features 235) from the spatial information.

[0046] The diagnostic tool may group a set of local features and a set of global features to obtain a diagnostic output. In certain embodiments, the diagnostic output includes one or more ophthalmic images (e.g., ophthalmic image 175). Each ophthalmic image may include a disease index (e.g., disease index 180). For example, the disease index may be in the form of one or more segmented regions within the ophthalmic image associated with an ophthalmic disease. The ophthalmic image may also be associated with a severity score (e.g., severity score 185) that identifies a particular ophthalmic disease along with an indication of the severity of the ophthalmic disease.

[0047] At block 520, the diagnostic tool transmits a representation of the ophthalmic information (e.g., the ophthalmic information) to a user (e.g., a clinician). The ophthalmic information may include at least a diagnostic output.

[0048] Although not shown in FIG. 5 , in certain embodiments, method 500 includes generating, by the diagnostic tool, a visualization of the plurality of spectral information based on the spectral information. In certain embodiments, the visualization includes a single interpretation image (e.g., interpretation image 125) having one or more features from across one or more spectral bands of the spectral information. In certain embodiments, the diagnostic tool generates the visualization of the plurality of spectral information further based on diagnostic criteria (e.g., diagnostic criteria 195) received from a user (e.g., a clinician) via a computing system (e.g., computing system 190). In certain embodiments, the ophthalmic information transmitted to the user (at block 520) also includes the visualization of the plurality of spectral information.

[0049] 6A-6B illustrate an exemplary sequence 600 for analyzing multiple spectral information to generate ophthalmic information, according to certain embodiments. As shown in FIG. 6A, the visualization tool 215 evaluates the spectral information 205, which includes multiple spectral frames 602, to generate an interpretation image 125. As shown, the interpretation image 125 is a single image that includes one or more (overlaid) features 604 from across the spectral frames 602.

[0050] 6B, the output from the local feature detection tool 220 and the output from the global feature detection tool 225 are used to generate an ophthalmic image 175 having a disease indicator 180. In the depicted embodiment, the disease indicator 180 is a segmented region of retinal disease (e.g., "hard exudates" caused by diabetic macular edema) and has a severity score of "5."

[0051] FIG. 7 illustrates an exemplary computing system 700 configured to automatically initiate image-guided surgery, according to certain embodiments. As shown, computing system 700 includes, without limitation, a processing unit 705, a network interface 715, memory 720, and storage 760, each connected to a bus 717. Computing system 700 may also include an I / O device interface 710 that connects I / O devices 712 (e.g., a keyboard, a display, and a mouse device) to computing system 700. Computing system 700 is generally under the control of an operating system (not shown). Examples of operating systems include the UNIX operating system, versions of the Microsoft Windows operating system, and distributions of the Linux operating system (UNIX is a registered trademark of The Open Group in the United States and / or other countries. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both). More generally, any operating system that supports the features disclosed herein may be used.

[0052] Processing unit 705 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). Processing unit 705 retrieves and executes programming instructions stored in memory 720 and storage 760. Bus 717 is used to transfer programming instructions and application data between processing unit 705, I / O device interface 710, storage 760, network interface 715, and memory 720. Note that processing unit 705 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, a single GPU, multiple GPUs, a single GPU with multiple processing cores, or any combination thereof. Memory 720 is included to generally represent random access memory. Storage 760 may be a disk drive or flash storage device. While shown as a single unit, storage 760 may be a combination of fixed or removable storage devices, such as a fixed disk drive, a removable memory card, optical storage, network-attached storage (NAS), or a storage area network (SAN), etc. Illustratively, memory 720 includes diagnostic tool 135, discussed in more detail above. Additionally, storage 760 includes input data 110 and ophthalmic information 120, also described above.

[0053] In summary, certain embodiments of the present disclosure provide a framework for applying image analysis and AI / ML techniques to automatically process and present clinically relevant information from data acquired using MSI / HSI techniques. The framework described herein can be used for a variety of ophthalmology applications, including disease detection, to overcome challenges in interpreting, visualizing, and utilizing MSI / HSI information.

[0054] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including a single element. As an example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or a, b, and c in any other order).

[0055] The foregoing description is provided to enable those skilled in the art to practice the various embodiments described herein. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Accordingly, the claims are not intended to be limited to the embodiments shown herein, but are to be accorded the full scope consistent with the claim language.

[0056] Reference to an element in the claims in the singular is intended to mean one or more, not one and only one, unless specifically stated otherwise. The term "some" refers to one or more, unless specifically stated otherwise. All structural and functional equivalents to the elements of the various embodiments described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated by reference herein and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended as a dedication to the public, regardless of whether such disclosure is expressly recited in the claims. No element of a claim is to be construed under 35 U.S.C. § 112(f) unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, the element is recited using the phrase "step for." The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

Claims

1. an imaging system adapted to capture a plurality of spectral information associated with the patient's eye; a memory containing executable instructions; a processor in data communication with the memory, extracting a first set of information from the plurality of spectral information and a second set of information from the plurality of spectral information, wherein the first set of information includes spectral information and the second set of information includes spatial information; generating a set of ophthalmic information associated with the eye of the patient based on evaluating the first set of information using a first deep learning model and the second set of information using a second deep learning model; Presenting a display of said set of ophthalmic information to a user. the processor configured to execute the executable instructions to An ophthalmology system comprising:

2. The ophthalmic system of claim 1 , wherein the plurality of spectral information comprises multispectral imaging (MSI) data.

3. The ophthalmic system of claim 1 , wherein the plurality of spectral information comprises hyperspectral imaging (HSI) data.

4. The ophthalmic system of claim 1 , wherein the processor is configured to execute the executable instructions to extract the second set of information by performing a dimensionality reduction operation on the plurality of spectral information.

5. The processor further comprises: generating a visualization of the plurality of spectral information based on evaluating the first set of information with a third deep learning model; Presenting the visualization to the user The ophthalmic system of claim 1 , configured to execute the executable instructions for:

6. the plurality of spectral information includes a plurality of images, each image associated with a different spectral band; the visualization includes a single image including multiple features from one or more of the multiple images. The ophthalmology system of claim 5 .

7. the plurality of spectral information includes a plurality of images, each image associated with a different spectral band; The processor is configured to execute the executable instructions to generate the set of ophthalmic information by (i) extracting, via the first deep learning model, a first set of features from one or more regions of one or more images of the plurality of images; and (ii) extracting, via the second deep learning model, a second set of features from the spatial information. The ophthalmic system of claim 1 .

8. the first set of features includes a set of local features; each of the one or more regions includes a single pixel from one of the multiple images; the second set of features includes a set of global features; The ophthalmic system of claim 7 .

9. The ophthalmic system of claim 7 , wherein the set of ophthalmic information includes a prediction of a condition associated with the eye of the patient.

10. The ophthalmology system of claim 9 , wherein the prediction includes (i) an ophthalmology image including one or more segmented regions associated with the condition, and (ii) a score indicating the severity of the condition.

Images