Generating multispectral imaging information using analytical multispectral imaging

Abstract

In certain embodiments, a system, computer-implemented method, and computer-readable medium are described for generating an analytical multispectral imaging (MSI) based on performing an MSI operation. The analytical MSI operation includes synchronizing scanning of a target by a broadband imaging device with generating a plurality of light signals directed toward the target from a plurality of broadband illumination sources. A set of spectral information associated with the target is generated based on scanning the target with the broadband imaging device. A set of spectral information associated with the target is generated based on performing a multispectral imaging (MSI) operation using at least the set of spectral information. Ophthalmic information is determined based on the set of MSI information.

Description

Background Art

[0001] Many microsurgical procedures require precise cutting and / or removal of various body tissues. For example, vitreoretinal surgeries (such as retinotomy, retinectomy, autologous retinal transplantation, and vitrectomy) often require cutting, removal, dissection, layering, coagulation, or other manipulation of intraocular tissues (such as the retina, vitreous humor, traction bands, and membranes).

[0002] Many of these intraocular tissues play a vital role in achieving vision. For example, the retina, or the innermost layer lining the back wall of the eye, is responsible for receiving, modulating, and transmitting visual stimuli from the external environment to the optic nerve and ultimately to the visual cortex of the brain. Structurally, the retina is a complex and delicate tissue with multiple types of cells arranged in multiple cell layers. Due to the retina's role in vision and its fragility, damage to it can result in severe vision loss or even permanent blindness. Therefore, cutting, removal, or other manipulation of the retina should be performed very carefully to avoid undesirable retinal trauma.

[0003] Some imaging systems may employ multispectral imaging (MSI) to provide the surgeon (and / or other clinicians) with information relating to spectral parameters of ocular tissue / structure. MSI is a technique that involves measuring (or capturing) light from a sample (e.g., ocular tissue / structure) at different wavelengths or spectral bands within the electromagnetic spectrum. MSI can capture more information from a sample that may not be visible through conventional imaging, which typically uses broadband illumination and broadband imaging sensors. In an exemplary MSI imaging system, the MSI information obtained by the MSI imaging system can be used to adjust the use of instruments (e.g., forceps, lasers, probes, etc.) used to manipulate ocular tissue / structure in real time during a surgical procedure.

[0004] However, currently, imaging systems must typically use highly specialized and expensive equipment in order to implement MSI. For example, in order to implement MSI, the illumination must typically be adjustable and / or the camera must be able to sense multiple spectral bands. As a result, conventional imaging systems typically employ highly specialized multispectral illumination sources and / or highly specialized multispectral cameras in order to implement MSI. However, given the high costs associated with surgical procedures, such highly specialized and expensive equipment may make it impractical to implement MSI in surgical procedures.

[0005] Therefore, it would be beneficial to provide improved systems, devices and methods for generating MSI to obtain ophthalmic information associated with various ocular tissues / structures. Summary of the invention

[0006] In certain embodiments, a system is provided. The system includes an illumination device, a broadband imaging device, a memory including executable instructions, and a processor in data communication with the memory. The illumination device includes a plurality of broadband illumination sources, and the broadband imaging device includes a plurality of imaging sensors. The processor is configured to execute the executable instructions to synchronize the scanning of a target by the broadband imaging device with the generation of a plurality of light signals directed to the target from the plurality of broadband illumination sources. The processor is further configured to execute the executable instructions to generate a set of spectral information associated with the target based on scanning the target with the broadband imaging device. The processor is further configured to execute the executable instructions to generate a set of MSI information associated with the target based on performing a multispectral imaging (MSI) operation using at least the set of spectral information. The processor is further configured to execute the executable instructions to determine ophthalmic information based on the set of MSI information.

[0007] In certain embodiments, a computer-implemented method is provided. The computer-implemented method includes synchronizing scanning of a target by a broadband imaging device with generating a plurality of light signals directed toward the target from a plurality of broadband illumination sources. The computer-implemented method also includes generating a set of spectral information associated with the target based on scanning the target with the broadband imaging device. The computer-implemented method also includes generating a set of multispectral imaging (MSI) information associated with the target based on performing a multispectral imaging (MSI) operation using at least the set of spectral information. The computer-implemented method further includes determining ophthalmic information based on the set of MSI information.

[0008] In certain embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium has computer-executable instructions stored thereon. The computer-executable instructions can be executed by one or more processors to perform operations. The operations include synchronizing scanning of a target by a broadband imaging device with generating multiple light signals directed to the target from multiple broadband illumination sources. The operations also include generating a set of spectral information associated with the target based on scanning the target with the broadband imaging device. The operations further include generating a set of multi-spectral imaging (MSI) information associated with the target based on performing a multi-spectral imaging (MSI) operation using at least the set of spectral information. The operations further include determining ophthalmic information based on the set of MSI information. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to be able to understand the above-mentioned features of the present disclosure in detail, the present disclosure briefly summarized above can be described in more detail with reference to the embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings only show exemplary embodiments and are therefore not to be considered as limiting the scope thereof, and other equally effective embodiments may be allowed.

[0010] Figure 1An example system for generating MSI information based on analytical MSI operations according to some embodiments is presented.

[0011] Figure 2 is a flow chart of a method for generating MSI information based on an analytical MSI operation according to certain embodiments.

[0012] Figure 3 An example of temporal intensity modulation of a lighting device according to some embodiments is shown.

[0013] Figure 4 is a flow chart of a method for performing analytical MSI operations according to certain embodiments.

[0014] Figure 5A A graph showing spectral emission functions according to certain embodiments is shown.

[0015] Figure 5B A graph showing a spectral sensitivity function according to some embodiments.

[0016] Figure 5C Shown is a graph of spectral basis functions generated as part of an analytical MSI operation according to certain embodiments.

[0017] Figure 5D An example of a simplified set of spectral basis functions generated as part of an analytical MSI operation in accordance with certain embodiments is shown.

[0018] Figure 5E Demonstrating some embodiments Figure 5D An example of band approximation for a simplified set of spectral basis functions in .

[0019] FIG. 6A to FIG. 6B A graph showing an example reconstruction of an initial spectrum using MSI information generated as part of an analytical MSI operation according to certain embodiments.

[0020] Figure 7 An example of using MSI information to determine ophthalmic information associated with a patient's eye is shown in accordance with certain embodiments.

[0021] Figure 8 Examples of using MSI information to visually enhance feature(s) of a patient's eye are shown in accordance with certain embodiments.

[0022] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation. DETAILED DESCRIPTION

[0023] Embodiments described herein provide ophthalmic systems, apparatuses, and techniques for generating MSI information associated with a patient's eye based on performing analytical MSI operations. In some embodiments, the analytical MSI operations are performed using non-MSI specialized equipment, such as (multiple) broadband illumination sources, (multiple) broadband imaging sensors (also referred to as (multiple) broadband cameras), etc., rather than using MSI specialized equipment, such as multi-spectral band illumination sources, multi-spectral band cameras, etc.

[0024] In certain embodiments described below, an analytical MSI operation involves assigning a reflectance spectrum of an imaging feature into multiple spectral bands based on one or more parameters of a non-MSI specialized device. Because a non-MSI specialized device may not be configured (or designed) to implement MSI (e.g., a broadband illumination device may not be adjustable, a broadband imaging device may not be able to sense multiple (e.g., 3 or more) spectral bands, etc.), certain embodiments use the parameter(s) of the non-MSI specialized device to generate an MSI approximation of the reflectance spectrum. The parameter(s) of the non-MSI specialized device may include the parameter(s) of the broadband imaging device and / or the parameter(s) of the broadband illumination device.

[0025] As part of generating the MSI approximation, certain embodiments use the parameter(s) of the broadband imaging device and the parameter(s) of the broadband illumination device to generate a set of spectral basis functions representing the combined operation of the broadband imaging device and the broadband illumination device. For example, the set of spectral basis functions may indicate one or more spectral sensitivities of the broadband imaging device at multiple operating wavelengths of the broadband illumination device. In the exemplary embodiments described below, the set of spectral basis functions is generated based on convolving the parameter(s) of the broadband illumination device (e.g., spectral emission function) with the parameter of the broadband imaging device (e.g., spectral sensitivity function).

[0026] Once the set of spectral basis functions representing the combined operation of the broadband illumination device and the broadband imaging device is generated, certain embodiments may perform spectral decomposition of the set of spectral basis functions to approximate the MSI of the reflectance spectrum obtained by the target. For example, through spectral decomposition, the reflectance spectrum may be assigned to a set of spectral bands based on the eigenvectors and eigenvalues ​​of the spectral basis functions. In this way, embodiments may use parameters of a non-MSI specialized device to approximate the MSI of the reflectance spectrum obtained from the target using a non-MSI specialized device.

[0027] An exemplary ophthalmic system described herein may include a broadband illumination device and a broadband imaging device. The broadband illumination device may include one or more broadband illumination sources, each of which may be independently modulated. For example, the intensity of each broadband illumination source may be independently controlled relative to the intensity (over time) of other broadband illumination sources of the broadband illumination device. In an exemplary embodiment, each broadband illumination source may be independently "turned on" or "turned off".

[0028] The broadband imaging device may include one or more broadband imaging sensors. For example, the broadband imaging device may be an RGB camera. Such an RGB camera may include an imaging sensor configured to capture light bands in the "red" spectral region of the visible spectrum, an imaging sensor configured to capture light bands in the "green" spectral region of the visible spectrum, and an imaging sensor configured to capture light bands in the "blue" spectral region of the visible spectrum.

[0029] In certain embodiments, the operation of the broadband illumination device may be synchronized with the operation of the broadband imaging device. For example, the temporal intensity modulation of the broadband illumination source(s) may be synchronized with the scanning of the target (e.g., eye tissue / structure) by the broadband imaging device. In an exemplary embodiment, the turning on / off of the broadband imaging sensor may be synchronized with the image frame of the broadband imaging device. In another exemplary embodiment, the turning on / off of the broadband imaging sensor may be synchronized with the scan line of the broadband imaging device.

[0030] In certain embodiments described in more detail below, an ophthalmic system may generate MSI information associated with a given target (e.g., ocular tissue / structure) based on performing an analytical MSI operation according to: (i) spectral information obtained by scanning the target with a broadband imaging device, (ii) parameter(s) of the broadband imaging device, and (iii) parameter(s) of a broadband illumination device. In an exemplary embodiment, the MSI information may include an MSI configuration in which the spectral information is assigned to a plurality of spectral bands. For example, based on the analytical MSI operation, the ophthalmic system may assign the spectral information to a plurality of spectral bands based on the parameter(s) of the broadband imaging device (e.g., the spectral sensitivity of the broadband imaging device) and the parameter(s) of the broadband illumination device (e.g., the spectral emission of the broadband illumination device).

[0031] In some embodiments, the MSI information generated by the analytical MSI operation can be considered to be substantially equivalent to the "initial" spectral information obtained by scanning the target with a broadband imaging device. That is, the MSI information and the initial spectral information can produce the same (or similar) sensor output(s) given the same (or similar) input(s).

[0032] In certain embodiments, the MSI information generated by the analytical MSI operation can be used to determine ophthalmic information about the patient's eye. The ophthalmic information can include, but is not limited to, diagnostic information, structural information, visual information, and / or functional information associated with the eye. In an exemplary embodiment, the ophthalmic information can include parameters of the ocular tissue / structure related to the physiological state and / or metabolic state of the ocular tissue. In another exemplary embodiment, the ophthalmic information can include enhanced visualization of the ocular tissue / structure. As used herein, the terms "information" and "data" can be used interchangeably to represent qualitative observations and / or quantitative data.

[0033] By providing analysis techniques for generating MSI information using non-MSI specialized equipment, the embodiments described herein enable ophthalmic systems with existing non-MSI specialized equipment to add functionality to physicians during ophthalmic surgery and exceed the capabilities of each individual hardware component by generating MSI information. Thus, the embodiments described herein can significantly reduce the costs associated with implementing MSI.

[0034] As used herein, MSI specialized equipment may refer to (multiple) devices that are specifically designed for MSI applications. Examples of such MSI specialized equipment may include, for example, multi-spectral band illumination sources (e.g., narrow-band illumination sources, narrow-band filters, etc.), multi-spectral band cameras (e.g., imaging sensors capable of sensing multiple spectral bands beyond the RGB spectral band), etc. In addition, as used herein, non-MSI specialized equipment may refer to (multiple) devices that are not specifically designed for MSI applications. Examples of such non-MSI specialized equipment may include, for example, broadband illumination sources, broadband illumination sources without narrow-band filters, broadband imaging cameras (e.g., RGB cameras), etc.

[0035] Figure 1 A system 100 for generating MSI information based on an analytical MSI operation according to certain embodiments is shown. The system 100 includes an ophthalmic system 130 and an intraocular illuminator 108 (also referred to as a probe) coupled to the ophthalmic system 130.

[0036] The intraocular illuminator 108 includes a handpiece 110 coupled to the proximal end of a shaft or "tube" 112, the handpiece 110 being configured to provide a user (e.g., an ophthalmic surgeon) with a graspable portion of the intraocular illuminator 108 to provide the surgeon with a means for manipulating the depth and position of the tube 112 within the eye 120 and for directing emitted light 150. The tube 112 is a substantially hollow stainless steel shaft or hypodermic tube that is configured to be inserted into the eye 120 via a cannula and a sclerotomy 140. In some examples, the tube 112 is fixedly coupled to the handpiece 110. In another example, the tube 112 is rotatable relative to the handpiece 110. Although the tube 112 is illustrated as a straight shaft, other embodiments include tubes 112 having other shapes. For example, a portion of the tube 112 may be curved to provide light to areas of the eye that are difficult to illuminate with a straight tube 112. In some embodiments, the intraocular illuminator 108 and its components are an instrument kit for ophthalmic surgery.

[0037] The intraocular illuminator 108 is further configured to house one or more optical fibers configured to direct light out of the distal end of the tube 112. For example, the optical fibers may include an optical fiber array (e.g., multiple optical fibers arranged in a regular linear arrangement or a 2-dimensional pattern) and / or a multi-core optical fiber (e.g., a single-mode (SM) or multi-mode (MM) optical fiber having multiple cores). In particular, the hollow portion of the tube 112 includes an internal compartment configured to house the optical fiber(s).

[0038] The ophthalmic system 130 includes an illumination device 104, a computing system 144 (also referred to as a controller), and an imaging device 102. The illumination device 104 is typically a broadband illumination device. For example, the illumination device 104 includes one or more (broadband) light sources 160 (also referred to as illumination sources). The (multiple) light sources 160 can generate illumination beams that can be used during ophthalmic surgery. For example, the (multiple) light sources 160 can generate (multiple) illumination beams alternatively, sequentially, or simultaneously. A user (such as a surgeon or surgical staff) can control the ophthalmic system 130 (e.g., via a foot switch, voice command, etc.) to emit illumination beams during ophthalmic surgery (such as vitreoretinal surgery).

[0039] The ophthalmic system 130 transmits an illumination beam from (multiple) light sources 160 to the intraocular illuminator 108 via an optical fiber 152. As shown, the handpiece 110 of the intraocular illuminator 100 is removably coupled to the distal end of the optical fiber 152, the proximal end of which is coupled to the illuminator 104. Note that in this document, the distal end of a component refers to the end closer to the patient's body or the end from which the illumination light is emitted from the illuminator. On the other hand, the proximal end of a component refers to the end facing away from the patient's body or close to, for example, a light source.

[0040] During surgery, the surgeon uses hand piece 110 to guide tube 112 into a patient's eye 120. Tube 112 is only partially inserted into eye 120, such that the proximal end of tube 112 is disposed outside of eye 120. Light source(s) 160 generate illumination beams 150 that illuminate the interior of eye 120, thereby allowing the interior to be viewed with an imaging device, such as imaging device 102.

[0041] As described above, in some embodiments, the (multiple) light sources 160 are (multiple) broadband light sources. For example, the (multiple) light sources 160 may include a first broadband light source 160-1 that emits light from a "red" spectral band of the visible spectrum, a second broadband light source 160-2 that emits light from a "blue" spectral band of the visible spectrum, and a third broadband light source 160-3 that emits light from a "green" spectral band of the visible spectrum. Each of the broadband light sources may be a light emitting diode (LED) light source. In some embodiments, the (multiple) light sources 160 are integrated with a console (not shown). In some other embodiments, the (multiple) light sources 160 are (multiple) independent light sources.

[0042] In certain embodiments described in more detail below, the intensity of each of the light sources 160 may be independently modulated over time. For example, a given light source 160 may be (i) "turned on" (or the intensity of the light source may be increased) or (ii) "turned off" (or the intensity of the light source may be decreased) at any given point in time. As described below, the temporal modulation of the intensity of each light source 160 may be controlled by the computing system 144, which is described in more detail below.

[0043] The system 100 uses (multiple) optical fibers 152 to transmit illumination light, and uses free space transmission for return light. For example, the illumination beam 150 is directed from the tube 112 onto tissue / structures inside the eye (e.g., the retinal surface), and the return light (e.g., at least a portion of the light reflected from the tissue / structures inside the eye) is collected by the imaging device 102. The imaging device 102 is a broadband imaging device (e.g., an RGB camera) having one or more imaging sensors 106. Each imaging sensor 106 can be configured to capture light from a specific spectral band of the visible spectrum. Assuming that the imaging device 102 is an RGB camera, the (first) imaging sensor 106-1 can be configured to capture a spectral band of light in the "red" spectral region of the visible spectrum, the (second) imaging sensor 106-2 can be configured to capture a spectral band of light in the "green" spectral region of the visible spectrum, and the (third) imaging sensor 106-3 can be configured to capture a spectral band of light in the "blue" spectral region of the visible spectrum.

[0044] The imaging device 102 is generally configured to generate (e.g., capture) an image(s) of a target tissue / structure (e.g., retinal surface) inside the eye based on the returned light. As used herein, the term "returned light" may include reflection, scattering, fluorescence, autofluorescence, Raman spectroscopy, or a combination thereof. In some embodiments, the operation of the imaging device 102 and the operation of the (multiple) light sources 160 may be synchronized (e.g., via the computing system 144). For example, the computing system 144 may synchronize the generation of an illumination beam (or signal) from the (multiple) light sources 160 with the pixel scanning of the (multiple) imaging sensor 106.

[0045] In an exemplary embodiment, the computing system 144 may synchronize the imaging device 102 and the light source(s) 160 so that the turning on / off of the light source(s) 160 is synchronized with each image frame captured by the imaging device 102. For example, the computing system 144 may control the imaging device 102 to capture three consecutive raw camera images (or readouts) of the patient's eye, wherein only one of the light sources 160 is turned on for each image. That is, in this example, each image frame may include return light from a single light source 160.

[0046] In another exemplary embodiment, the computing system 144 may synchronize the imaging device 102 and the light source(s) 160 such that the turning on / off of the light source(s) 160 is synchronized with the scan lines (or rolling shutter) of the imaging device 102. In this embodiment, within each image frame captured by the imaging device 102, a given scan line will include return light from a single light source 160. That is, only one of the light sources 160 is turned on for each scan line of the image frame.

[0047] Notice, Figure 1 Reference examples of the ophthalmic system 130 are shown, and in other embodiments, the ophthalmic system 130 may have a different configuration. For example, while Figure 1 The imaging device 102 and the illumination device 104 are depicted within the ophthalmic system 130, but in some embodiments, the imaging device 102 and / or the illumination device 104 may be separate from the ophthalmic system 130. As a reference example, the illumination device 104 may be included within a surgical console, and the imaging device 102 may be part of a digital microscope that includes optical components (e.g., an objective lens) and a digital camera (e.g., an RGB camera) to output images.

[0048] The display 142 is coupled to the computing system 144 and is capable of displaying MSI information and / or ophthalmic information associated with the eye 120 to the surgeon, including MSI information generated by performing analytical MSI operations with the imaging device 102 and the illumination device 104. In the illustrated embodiment, the display 142 is separate from the imaging device 102 and the ophthalmic system 130. In some other embodiments, the display 142 is integral to the imaging device 102 or the ophthalmic system 130. In some embodiments, the display 142 includes an augmented reality display. In some embodiments, the display 142 includes a virtual reality display. In some embodiments, the display 142 includes a three-dimensional display to provide depth information to the surgeon. In some embodiments, the display 142 receives image(s) captured by the imaging device 102 and presents the image(s) for viewing. In some embodiments, the display 142 receives MSI information from the computing system 144 and presents the MSI information for viewing.

[0049] The computing system 144, such as a programmable computer, is generally configured to perform one or more of the operations described herein for performing analytical MSI operations. The computing system 144 is coupled to one or more of the illumination device 104, the imaging device 102, and the display 142. For example, the computing system 144 may control the operation of the ophthalmic system 130 using direct control of the illumination device 104, the imaging device 102, and / or the display 142 or using indirect control of other controllers associated therewith. During surgery, the computing system 144 may enable data collection and feedback from the respective components to coordinate the operation of the ophthalmic system 130.

[0050] The computing system 144 includes a central processing unit (CPU) 132, a memory 134 operable with the CPU 132, and circuit(s) 136. The circuit(s) 136 are conventionally coupled to the CPU 132 and include caches, clock circuits, input / output subsystems, power supplies, etc., and combinations thereof, coupled to various components of the ophthalmic system 130.

[0051] In certain embodiments, the CPU 132 is one of any form of general purpose computer processor used in industrial facilities, such as a programmable logic controller (PLC), for controlling various monitoring system components and subprocessors. The memory 134 coupled to the CPU 132 is typically one or more of readily available memories, including volatile or non-volatile memory. For example, the memory 134 may be a local or remote random access memory (RAM), read-only memory (ROM), a floppy disk drive, a hard disk, or any other form of digital storage device.

[0052] Herein, the memory 134 stores instructions that, when executed by the CPU 132, facilitate the operation of the ophthalmic system 130. The instructions in the memory 134 are in the form of a program product, such as a program (e.g., a middleware application, a device software application, etc.) that implements the methods of the present disclosure. As shown, the memory 134 includes an MSI component 138 (e.g., a software component) that is generally configured to perform analytical MSI operations to generate MSI information. The MSI component 138 is described in more detail below. Note that the methods disclosed herein can be performed using one or more of the provided ophthalmic system embodiments, and therefore, the ophthalmic system 130 is described in the following examples for illustration purposes only.

[0053] Figure 2 is a flow chart of a method 200 for generating MSI information based on an analytical MSI operation according to one embodiment. The method 200 may be performed by one or more components of a system (eg, system 100).

[0054] The method 200 enters at block 202 where the system synchronizes generation of one or more light signals from an illumination device (e.g., illumination device 104) with operation of an imaging device (e.g., imaging device 102). As described above, in certain embodiments, temporal modulation of the intensity of light source(s) within the illumination device (e.g., light source 160) may be synchronized with captured image frame(s) of the imaging device or with scan line(s) within a captured image frame of the imaging device.

[0055] For example, briefly refer to Figure 3 , which illustrates an example of temporal modulation of the intensity of multiple light sources 160 according to some embodiments. In the depicted embodiment, the imaging sensor 106 is a 1 / 3 inch 3.1 megapixel (Mp) CMOS digital image sensor having an active pixel array 304 (e.g., a 2048 (H)×1536 (V) pixel array). The pixel array may include RGB pixels. However, note that Figure 3 The pixel arrays depicted in are examples only, and other configurations / sizes of pixel arrays may be used consistent with the functionality described herein.

[0056] The pixel array 304 has a coarse integration time defined by the number of row cycles (TROW) between row reset and row read. The row cycle (TROW) is defined as the time between row read operations. The pixel array 304 includes a vertical blanking signal 308 and a horizontal blanking signal 310. Each vertical blanking signal 308 is configured as a blanking row (of the pixel array 304) to indicate when a new image frame begins. The horizontal blanking signal 310 is configured as a blanking column (across one or more rows) to indicate when a new scan line begins. Therefore, in the depicted example, the imaging sensor 106 can start the frame readout from a vertical blanking row (e.g., the vertical blanking signal 308), followed by one or more active rows (between the vertical blanking rows 308). Each active row of the frame readout can start from the horizontal blanking signal 310. That is, the imaging sensor 106 can read the first vertical blanking row at the beginning of the frame cycle and read the last active row at the end of the row cycle. Configuring pixel array 304 in the manner described above is beneficial because it allows imaging sensor 106 to capture continuous video and single frames and operate in different modes, including, for example, a high dynamic range operating mode in which imaging sensor 106 can capture a wide range of illumination with intensities varying over a 100 dB range or more.

[0057] In certain embodiments, the modulation of the intensity of the light source 160 is synchronized with the line (and frame) signals (e.g., the vertical blanking signal 308 and the horizontal blanking signal 310) of the imaging sensor 302. For example, the intensity of the light source 160 can be controlled by multiplexing synchronized with the vertical blanking signal 308 and the horizontal blanking signal 310. As described above, in an exemplary embodiment, the modulation method can involve turning "on" and "off" a particular light source 160 within a particular row. The benefit of multiplexing within a single image frame is that spectral information in a local area under different light intensity combinations can be compared without the need to correct for eye movement. Rapid modulation of light intensity can also minimize visualization interruptions during ophthalmic surgery.

[0058] like Figure 3As shown, the modulation of the intensity of the light sources 160 is controlled so that only one of the light sources 160 is "turned on" for the duration of the corresponding scan line 306. For example, when the scan line 306 is on the "n"th row, the light source 160-1 is "turned off", the light source 160-2 is "turned off", and the light source 160-3 is "turned on"; when the scan line 306 is on the "n+1"th row, the light source 160-1 is "turned off", the light source 160-2 is "turned on", and the light source 160-3 is "turned off"; when the scan line 306 is on the "n+2"th row, the light source 160-1 is "turned on", the light source 160-2 is "turned off", and the light source 160-3 is "turned off"; and so on. By modulating the intensity of the (multiple) light sources 160 in this manner, the ophthalmic system 130 can differentially sample spectral bands in the return light from the patient's eye.

[0059] In some embodiments, the intensity of light source 160 can be modulated based on the percentage of "off" time. Doing so can maintain a relatively constant average illumination and thereby minimize visual flicker in the visual output. Additionally, in some embodiments, the modulation of the intensity of light source 160 can be limited to a target area of ​​the frame, such as the center of the frame.

[0060] Return to reference Figure 2 , at block 204, the system captures multiple images of a target (e.g., eye tissue / structure) via an imaging device. Each image includes spectral information corresponding to at least one of the (multiple) light signals reflected from the target. For example, for a given target (or feature) in a patient's eye, the following equation (1) can be used to describe the (multiple) light signals reflected from the target and captured by the imaging device. (Note that for clarity, the following equations are described for 3 light sources and 3 types of image sensors; however, the following equations can be used for any number of light sources and / or any number of image sensors):

[0061] I 传感器 =Σ λ Sensor (λ) · Lighting (λ) · Reflectivity (λ) (1)

[0062] Where, for three independently modulated primary light sources 160, reflectivity (λ) is the spectral reflectivity of the target at wavelength (λ), and the spectral sensitivity of the imaging sensor is [sensor R ,sensor G ,sensor B ], and the spectral emission of the lighting device 104 is [lighting 1, lighting 2, lighting 3].

[0063] In some embodiments, with the temporal intensity modulation method described herein (e.g., where only one primary light source in the illumination device is turned on at a time), the readout from the imaging device can be expressed using the following equation (2), which can be expressed in matrix form using equation (3). Note that the summation sign is omitted in equation (2) for clarity.

[0064]

[0065] I 相机 = basis matrix × reflectivity (3)

[0066] Here, equation (4) is used to express I 相机 , and use equation (5) to express the basis matrix:

[0067]

[0068]

[0069] At block 206, the system performs an analytical MSI operation based at least in part on the plurality of images to generate MSI information. In some embodiments, the system may generate the MSI information based on spectral information corresponding to at least one of a light signal reflected from the target, a parameter of the imaging device (e.g., a spectral sensitivity of the imaging sensor), and a parameter of the lighting device (e.g., a spectral emission of the lighting device). In some embodiments, the parameter of the imaging device and / or the parameter of the lighting device may be pre-configured parameters associated with the imaging system.

[0070] In some embodiments, the MSI information generated by the analytical MSI operation may include an MSI configuration of the target surface reflectance. For example, the MSI configuration may assign (or express) the target surface reflectance as a multi-band approximation, an example of which is shown in (6):

[0071]

[0072] Given a multi-band approximation (e.g., spectral bands), the surface reflectance can be approximated using equation (7):

[0073]

[0074] (Note that the spectral band T is the transpose of the matrix spectral bands).

[0075] In certain embodiments, using the approximation in (7), the following equation (8) can be obtained and used for analytical MSI operation:

[0076] I 相机 = Basis matrix × spectral band T× I MSI (8)

[0077] in, is the intensity within each MSI spectral band.

[0078] In some embodiments, the spectral bands used for analytical MSI (eg, IMSI) may be selected so that base matrix × spectral band T Reversible. For example, let the basis matrix × spectral band T Reversibility allows the conversion from an RGB image captured by an imaging device to a plurality of MSI images to be completely reversible. In this case where the conversion from an RGB image captured by an imaging device to a plurality of MSI images is completely reversible, the MSI information generated by the analytical MSI operation can be used to visually enhance certain features of the patient's eye. For example, image enhancement (e.g., specific "blue" signal enhancement) can be implemented in a multi-spectral domain for one spectral band using spectral information obtained by scanning multiple lines in a frame (or by scanning multiple frames) while modulating the intensity of the light source differently. The image enhancement can then be converted back to an RGB image to enhance visualization of surgery (e.g., blue light in vitrectomy).

[0079] Additionally, in certain embodiments, as part of the analytical MSI, a singular value decomposition operation of a basis matrix may be performed. A singular value decomposition operation may be performed on the basis matrix to determine the number of bands of the MSI information. For example, the number of bands of the MSI information may be set equal to the column space of the basis matrix (determined according to the singular value decomposition operation). Because the number of singular values ​​of the column space of the basis matrix (of the first set of column vectors) may be significantly greater than the remaining singular values ​​(of the second set of column vectors), the analytical MSI may be defined as having a number of bands equal to the number of the first set of column vectors (e.g., the column space of the basis matrix). The number of bands may then have a cut point defined to match the spectral shape of the first set of column vectors. Note that the following description of Figure 4 The analytical MSI procedure is described in more detail.

[0080] At block 208, the system transmits the MSI information to a computing device for display and / or further processing. As described above, in certain embodiments, the MSI information can be used to enable more anatomical and functional assessments of the eye condition. For example, the MSI information can be used to identify pathological states of eye tissue using spectral characteristics and distinguish them from healthy eye tissue, to infer the metabolic state of the retina using oxygen content in the blood (assuming that the spectral reflectance of blood vessels varies with oxygen content levels), to enhance visualization of eye structures, etc.

[0081] Figure 44 is a flow chart of a method 400 for performing analytical MSI operations according to one embodiment. The method 400 may be performed by one or more components of a system (e.g., system 100). In some embodiments, the method 400 may be performed to implement Figure 2 The operations in block 206 of method 200 shown in FIG.

[0082] The method 400 enters at block 402, where the system generates a first set of spectral basis functions (e.g., the basis functions in equation (5)) based on one or more parameters of an imaging device (e.g., imaging device 102) and one or more parameters of an illumination device (e.g., illumination device 104). As described above, the (multiple) parameters of the imaging device may include the spectral sensitivity of an imaging sensor (e.g., imaging sensor 106) of the imaging device. The (multiple) parameters of the illumination device may include the spectral emission of a light source (e.g., light source 160) of the illumination device.

[0083] In some embodiments, the first set of spectral basis functions may be generated by performing a convolution of (i) the spectral sensitivity of the imaging sensor and (ii) the spectral emission of the light source. FIG. 5A to FIG. 5C , for example, shows Figure 5A The three spectral emission functions 5101-3 in the graph 502 and Figure 5B 5) is an exemplary convolution of three spectral sensitivity functions 512 1-3 in graph 504 of FIG. In graph 502, function 510-1 (e.g., illumination 1 (λ) in equation (5)) is the spectral emission of a "red" broadband light source, function 510-2 (e.g., illumination 2 (λ) in equation (5)) is the spectral emission of a "green" broadband light source, and function 510-3 (e.g., illumination 3 (λ) in equation (5)) is the spectral emission of a "blue" broadband light source. In graph 504, function 512-1 (e.g., sensor 512 in equation (5)) is the spectral emission of a "red" broadband light source, function 510-2 (e.g., illumination 2 (λ) in equation (5)) is the spectral emission of a "green" broadband light source, and function 510-3 (e.g., illumination 3 (λ) in equation (5)) is the spectral emission of a "blue" broadband light source. R (λ)) is the spectral sensitivity of the “red” imaging sensor, function 512-2 (e.g., the sensor in equation (5) G (λ)) is the spectral sensitivity of the “green” imaging sensor, and function 512-3 (the sensor in equation (5) B (λ)) is the spectral sensitivity of the “blue” imaging sensor.

[0084] In some embodiments, the set of basis functions may be generated by convolving each function 510 with each function 512. Figure 5CAs shown in the diagram 506 of FIG. 506 , for example, basis function 514-1 is generated based on convolution of function 510-1 and function 512-1, basis function 514-2 is generated based on convolution of function 510-2 and function 512-1, basis function 514-3 is generated based on convolution of function 510-3 and function 512-1, basis function 514-4 is generated based on convolution of function 510-1 and function 512-2, basis function 514-5 is generated based on convolution of function 510-3 and function 512-1, Basis function 514-6 is generated based on convolving function 510-3 and function 512-2, basis function 514-7 is generated based on convolving function 510-1 and function 512-3, basis function 514-8 is generated based on convolving function 510-2 and function 512-3, and basis function 514-9 is generated based on convolving function 510-3 and function 512-3.

[0085] Return to reference Figure 4 At block 404, the system generates a second set of spectral basis functions based on performing spectral decomposition of the first set of spectral basis functions. In some embodiments, the number of spectral basis functions in the second set of spectral basis functions can be less than the number of spectral basis functions in the first set of spectral basis functions. Figure 5D , for example, shows an exemplary spectral decomposition of a set of basis functions. In the depicted embodiment, Figure 5C The set of basis functions in the diagram 506 of FIG. 506 is simplified to Figure 5D 506 . A set of basis functions in graph 508 of FIG. 506 (which includes spectral basis functions 516 1-5). In this example, the set of basis functions is reduced from 9 to 5 based on the eigenvalues ​​of the 9 basis functions. Note that graph 508 shows the column space of the spectral basis functions from graph 506 .

[0086] continue Figure 4 At block 406, the system determines a corresponding band approximation for each spectral basis function in the second set of spectral basis functions. For example, the system may determine a multi-band approximation to use as the spectral band in equation (7) T At block 408, the system estimates a reflectance spectrum of the target based at least in part on the corresponding band approximation. For example, the operation in block 408 may be performed using equation (7).

[0087] At block 410, the system determines whether the reflection spectrum satisfies a predetermined condition. For example, in some embodiments, the reflection spectrum determined in block 406 (e.g., the reflection spectrum approximated by the MSI) should be equivalent to the initial reflection spectrum (e.g., the spectrum of the returned light from the target feature) with a certain amount of reconstruction error tolerance. For example, Fig. 6A and Figure 6BGraph 610 and graph 620 show, respectively, an MSI reconstruction of an initial spectrum (eg, using a defined set of MSI bands) and a full reconstruction of an initial spectrum (eg, using a full set of spectral basis functions).

[0088] Return to reference Figure 4 , if the difference between the reflectance spectrum and the initial reflectance spectrum is greater than a threshold value (or outside a threshold range), the system can proceed to block 406 to determine another corresponding band approximation for each spectral basis function in the second set of spectral basis functions. On the other hand, if the reflectance determined in block 406 is equal to the initial reflectance spectrum, the system transmits an indication of the band approximation (block 412) to the computing system for display and / or further processing. In some embodiments, the system can also transmit an indication of the reflectance spectrum to the computing system. In some embodiments, the system itself can display the band approximation and / or perform further processing on the band approximation instead of transmitting the indication of the band approximation at block 412.

[0089] Figure 5E An example of band approximations is shown in graph 520 of . As shown, graph 520 includes a first approximation 518-1 of a band of spectral basis function 516-1, a second approximation 518-2 of a band of spectral basis function 516-2, a third approximation 518-3 of a band of spectral basis function 516-3, a fourth approximation 518-4 of a band of spectral basis function 516-4, and a fifth approximation 518-5 of a band of spectral basis function 516-5. Figure 5E In , each cut point of the corresponding band approximation can be aligned with the waveform of its corresponding spectral basis function.

[0090] As described above, in certain embodiments, the MSI information generated using the analytical MSI techniques described herein can be used to determine ophthalmic information. Figure 7 An example of using MSI information to determine the blood oxygen content of the eye according to some embodiments is shown. As shown, assuming that the MSI information includes five spectral bands obtained from the column space of nine spectral basis functions, two of the five spectral bands can be used to calculate the blood oxygen content. In another example, Figure 8 The use of MSI information to perform image enhancement is shown. In this example, the image enhancement is performed in the multispectral domain and includes visualization of a subset of the MSI information (eg, band approximation 518-1 is enhanced relative to other band approximations 518 2-4).

[0091] In summary, embodiments of the present disclosure enable the generation of MSI information based on the use of non-MSI specialized equipment (such as broadband illumination devices and broadband imaging devices) to perform analytical MSI techniques without using MSI specialized equipment. The ophthalmic systems and / or methods described herein are particularly advantageous for significantly reducing the costs associated with implementing MSI.

[0092] As used herein, the phrase "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover any combination of a, b, c, ab, ac, bc, and abc, as well as multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc or any other order of a, b, and c).

[0093] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the claims are not intended to be limited to the embodiments shown herein, but are to be given the full scope consistent with the language of the claims.

[0094] In the claims, unless otherwise specified, reference to a singular element is not intended to mean "one and only one", but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, regardless of whether such disclosure is explicitly stated in the claims, the content disclosed herein is not intended to be dedicated to the public. According to 35 U.S.C. §112 (f), the elements of any claim will not be interpreted unless the phrase "device for..." is used to expressly state these elements, or in the case of a method claim, the phrase "step for..." is used to state these elements. The word "exemplary" is used herein to mean "used as an example, instance, or illustration". Any aspect described as "exemplary" herein is not necessarily interpreted as being preferred or advantageous over other aspects.

Claims

1. A system comprising: an illumination device comprising a plurality of broadband illumination sources; A broadband imaging device, the broadband imaging device comprising a plurality of imaging sensors; a memory including executable instructions; and a processor in data communication with the memory and configured to execute the executable instructions to perform the following operations: synchronizing scanning of a target by the broadband imaging device with generation of a plurality of optical signals directed toward the target from the plurality of broadband illumination sources; generating a set of spectral information associated with the target based on scanning the target with the broadband imaging device; generating a set of multi-spectral imaging (MSI) information associated with the target based on performing a multi-spectral imaging (MSI) operation using at least the set of spectral information; as well as Ophthalmic information is determined based on the set of MSI information.

2. The system of claim 1, wherein: The processor is configured to execute the executable instructions to synchronize the scanning of the target with the generation of the plurality of light signals by synchronizing the modulation of the intensity of each of the plurality of broadband illumination sources with the scanning of the target.

3. The system of claim 2, wherein: The scanning of the target is synchronized with the modulation of the intensity of each of the plurality of broadband illumination sources such that each image frame captured by the broadband imaging device includes return light from only one of the plurality of light signals.

4. The system of claim 2, wherein: The scanning of the target is synchronized with the modulation of the intensity of each of the plurality of broadband illumination sources such that each scan line of an image frame captured by the broadband imaging device includes return light from only one of the plurality of light signals.

5. The system of claim 1, wherein: The processor is configured to execute the executable instructions to perform the MSI operation by: generating a first set of spectral basis functions based on one or more first parameters of the illumination device and one or more second parameters of the broadband imaging device; generating a second set of spectral basis functions based on the spectral decomposition of the first set of spectral basis functions; as well as A corresponding band approximation is determined for each spectral basis function in the second set of spectral basis functions.

6. The system of claim 5, wherein: The number of spectral basis functions in the second set of spectral basis functions is smaller than the number of spectral basis functions in the first set of spectral basis functions.

7. The system of claim 5, wherein: The processor is configured to execute the executable instructions to further perform the MSI operation by: generating a reconstruction of the spectral information using the band approximation; and A reconstruction of the spectral information is generated for comparison with the spectral information.

8. The system of claim 7, wherein: The processor is configured to execute the executable instructions to perform the MSI operation by further determining another corresponding band approximation value for each spectral basis function in the second set of spectral basis functions when the comparison satisfies a predetermined condition.

9. The system of claim 7, wherein: The processor is configured to execute the executable instructions to transmit the MSI information for at least one of display or processing when the comparison satisfies a predetermined condition.

10. The system of claim 1, wherein: The ophthalmic information includes one or more parameters of ocular tissue, which are related to the physiological state of the ocular tissue.

11. The system of claim 1, wherein: The ophthalmic information includes a visualization of a subset of the MSI information.

12. A computer-implemented method comprising: synchronizing scanning of a target by a broadband imaging device with generation of a plurality of optical signals directed toward the target from a plurality of broadband illumination sources; generating a set of spectral information associated with the target based on scanning the target with the broadband imaging device; generating a set of multi-spectral imaging (MSI) information associated with the target based on performing a multi-spectral imaging (MSI) operation using at least the set of spectral information; as well as Ophthalmic information is determined based on the set of MSI information.

13. The computer-implemented method of claim 12, wherein: Synchronizing the scanning of the target with the generation of the plurality of light signals includes synchronizing the modulation of the intensity of each of the plurality of broadband illumination sources with the scanning of the target.

14. The computer-implemented method of claim 13, wherein: The scanning of the target is synchronized with the modulation of the intensity of each of the plurality of broadband illumination sources such that each image frame captured by the broadband imaging device includes return light from only one of the plurality of light signals.

15. The computer-implemented method of claim 13, wherein: The scanning of the target is synchronized with the modulation of the intensity of each of the plurality of broadband illumination sources such that each scan line of an image frame captured by the broadband imaging device includes return light from only one of the plurality of light signals.

16. The computer-implemented method of claim 12, wherein: Executing the MSI operation includes: generating a first set of spectral basis functions based on one or more first parameters of the illumination device and one or more second parameters of the broadband imaging device; generating a second set of spectral basis functions based on the spectral decomposition of the first set of spectral basis functions; and A corresponding band approximation is determined for each spectral basis function in the second set of spectral basis functions.

17. The computer-implemented method of claim 16, wherein: The number of spectral basis functions in the second set of spectral basis functions is smaller than the number of spectral basis functions in the first set of spectral basis functions.

18. The computer-implemented method of claim 16, wherein: Executing the MSI further comprises: generating a reconstruction of the spectral information using the band approximation; and A reconstruction of the spectral information is generated for comparison with the spectral information.

19. The computer-implemented method of claim 18, wherein: Performing the MSI operation further includes determining another corresponding band approximation for each spectral basis function in the second set of spectral basis functions when the comparing satisfies a predetermined condition.

20. A non-transitory computer readable medium having computer executable instructions stored thereon, the computer executable instructions being executable by one or more processors to perform operations comprising: synchronizing scanning of a target by a broadband imaging device with generation of a plurality of optical signals directed toward the target from a plurality of broadband illumination sources; generating a set of spectral information associated with the target based on scanning the target with the broadband imaging device; generating a set of multi-spectral imaging (MSI) information associated with the target based on performing a multi-spectral imaging (MSI) operation using at least the set of spectral information; as well as Ophthalmic information is determined based on the set of MSI information.

Images