Systems and methods for guiding an imaging device to detect fluorescence and determine a lifetime of the fluorescence

Abstract

An example fluorescence imaging control system can direct (802) an illumination source to illuminate a scene with fluorescence excitation illumination configured to excite a fluorophore present at the scene; and direct (804) an imaging device to detect fluorescence emitted by the fluorophore in response to excitation of the fluorophore by the fluorescence excitation illumination. The fluorescence imaging control system can determine (806) a lifetime of the fluorescence based on the detected fluorescence. The fluorescence imaging control system can determine (808) an identity of the fluorophore based on the determined lifetime of the fluorescence.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 889,433, filed August 20, 2019, entitled “FLUORESCENCE IMAGING CONTROLSYSTEMS AND METHODS,” the contents of which are incorporated herein by reference in their entirety. Background Technology

[0003] Imaging devices (e.g., endoscopes) can be used during surgical procedures to capture images of the surgical area associated with the patient. These images can be presented to the surgeon during the procedure (e.g., as a video stream) to assist the surgeon in performing the procedure. In some cases, the image of the surgical area may include other captured images, such as fluorescence images, or images enhanced with other captured images. Fluorescence images are generated based on the fluorescence emitted by fluorophores upon excitation by a light source. For example, a fluorescence image can be used to highlight certain portions of the surgical area with a selected color (e.g., green).

[0004] Many fluorophores are available for use, and new fluorophores are constantly being developed. However, there is still room for improvement and expansion in the use of fluorescence imaging in medical imaging. Summary of the Invention

[0005] The following description presents a simplified summary of one or more aspects of the methods and systems described herein in order to provide a basic understanding of these aspects. This summary is not a broad overview of all anticipated aspects and is neither intended to identify all principal or key elements of all aspects nor to outline the scope of any or all aspects. Its sole purpose is to present, in a simplified form, some concepts of one or more aspects of the methods and systems described herein as a prelude to the more detailed description presented below.

[0006] An exemplary system may include a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute instructions to: direct an illumination source to illuminate a scene with fluorescence excitation illumination configured to excite fluorophores present in the scene; direct an imaging device to detect fluorescence emitted by the fluorophores in response to excitation of the fluorophores by the fluorescence excitation illumination; determine the lifetime of the fluorescence based on the detected fluorescence; and determine the identity of the fluorophores based on the determined lifetime of the fluorescence.

[0007] Another exemplary system may include a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute instructions to: guide an illumination source to illuminate a scene with fluorescence excitation illumination configured to excite fluorophores present in tissue at the scene; guide an imaging device to detect fluorescence emitted by the fluorophores in response to excitation of the fluorophores by the fluorescence excitation illumination; determine the lifetime of the fluorescence based on the detected fluorescence; and determine the type of tissue in which the fluorophores are present based on the determined lifetime of the fluorescence.

[0008] An exemplary method may include: guiding an illumination source to illuminate a scene with fluorescence excitation illumination via a fluorescence imaging control system, the fluorescence excitation illumination being configured to excite fluorophores present at the scene; guiding an imaging device to detect fluorescence emitted by the fluorophores in response to excitation of the fluorophores by the fluorescence excitation illumination via the fluorescence imaging control system; determining the lifetime of the fluorescence based on the detected fluorescence via the fluorescence imaging control system; and identifying the fluorophores based on the determined lifetime of the fluorescence via the fluorescence imaging control system. Attached Figure Description

[0009] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of this disclosure. Throughout the drawings, the same or similar reference numerals denote the same or similar elements.

[0010] Figure 1 An exemplary imaging system based on the principles described herein is shown.

[0011] Figure 2 An exemplary fluorescence imaging control system based on the principles described herein is shown.

[0012] Figure 3 Exemplary fluorescence excitation and illumination curves and fluorescence decay curves, as well as exemplary timing sequences for emission fluorescence excitation and illumination and activation fluorescence detection, are shown in accordance with the principles described herein.

[0013] Figure 4A An exemplary multi-region photodetector based on the principles described herein is shown.

[0014] Figure 4B Activation based on the principles described herein is shown. Figure 4A An exemplary timing sequence for a multi-region photodetector.

[0015] Figure 5 An exemplary enhanced medical image is shown based on the principles described herein.

[0016] Figure 6A and Figure 6B An exemplary fluorescence image based on the principles described herein is shown.

[0017] Figure 7 An exemplary computer-assisted surgical system based on the principles described herein is shown.

[0018] Figure 8 An exemplary method for determining the identity of a fluorophore present in a scene, based on the principles described herein, is shown.

[0019] Figure 9 An exemplary method is shown for identifying one or more regions of a tissue of a distinct tissue type in which fluorophores are present, based on the principles described herein.

[0020] Figure 10 An exemplary computing device based on the principles described herein is shown. Detailed Implementation

[0021] This paper describes a fluorescence imaging control system and method. As will be described in more detail below, the fluorescence imaging control system can be configured to guide an illumination source to illuminate a scene with fluorescence excitation illumination, the fluorescence excitation illumination being configured to excite one or more fluorophores present at the scene, and to guide an imaging device to detect fluorescence emitted by the fluorophores(one or more) in response to the excitation of the fluorophores(one or more) by the fluorescence excitation illumination. The fluorescence imaging control system can also be configured to determine the lifetime of fluorescence based on the detected fluorescence. Based on the determined lifetime of fluorescence, the fluorescence imaging control system can determine the identity of the fluorophores or identify the tissue type or region of different tissue types in which the fluorophores are present.

[0022] As used herein, “fluorescence lifetime” (also referred to herein as “lifetime”) can refer to the amount of time (e.g., average amount) a fluorophore molecule remains in an excited state before returning to its ground state by emitting photons. When a fluorophore molecule absorbs a photon of appropriate energy (e.g., fluorescence excitation illumination with a specific wavelength), the molecule transitions to an excited state. The excited fluorophore molecule then returns to its ground state through a series of decay processes, one of which involves the spontaneous emission of photons at a decay rate of k. The fluorescence lifetime τ is inversely proportional to the decay rate k (τ = 1 / k). If a group of fluorophores is excited, the fluorescence lifetime can also be expressed as the time it takes for N excited molecules to decrease to 1 / e of the original. That is, the fluorescence lifetime is the time required for N excited molecules to decrease exponentially to N / e (~36.8%) of the original group. Assuming that the intensity of fluorescence is related to the group of fluorophores returning to the ground state, the fluorescence will decay over time according to the following equation [1]:

[0023] I(t) = I0e -t / τ [1]

[0024] Where t is time, τ is the fluorescence lifetime, I0 is the initial intensity of fluorescence at t=0, and I(t) is the intensity at time t.

[0025] In some examples, a fluorescence imaging control system can be configured to perform one or more operations based on the determined identity of the fluorophore. For example, the fluorescence imaging control system can configure the fluorescence excitation illumination (e.g., wavelength or band, waveform, intensity, frequency, pulse width, period, modulation, etc.) and / or the imaging device (e.g., sampling rate, exposure time, gain, activation timing, etc.) based on the determined identity of the fluorophore (e.g., based on the optical properties of the identified fluorophore). Additionally or alternatively, the fluorescence imaging control system can configure the display of images generated based on detected fluorescence (e.g., fluorescence images, enhanced images, etc.) based on the identity of the fluorophore. These and other operations that can be performed by the fluorescence imaging control system based on the determined identity of the fluorophore are described herein.

[0026] In an additional example, the fluorescence imaging control system can be configured to perform one or more operations based on a determined type of tissue in which fluorophores are present. For example, the fluorescence imaging control system can configure the fluorescence excitation illumination and / or the imaging device based on a determined type of tissue in which fluorophores are present. Additionally or alternatively, the fluorescence imaging control system can configure the display of images generated based on detected fluorescence (e.g., fluorescence images, enhanced images, etc.) based on the type of tissue in which fluorophores are present. These and other operations that can be performed by the fluorescence imaging control system based on a determined type of tissue in which fluorophores are present are described herein.

[0027] The systems and methods described herein can offer various benefits. For example, they can automatically identify fluorophores present in patient tissues without requiring user input indicating the fluorophore's identity. In particular, the systems and methods described herein can be configured to identify fluorophores based on the measured lifetime of detected fluorescence. This can advantageously lead to the determination of the identity and concentration of fluorophores present in patient tissues essentially in real time. Furthermore, determining the fluorescence lifetime emitted by fluorophores from different tissue types enables surgical scene segmentation, delineation of critical tissues, such as, but not limited to, the nervous and vascular systems, and healthy versus diseased (e.g., cancerous) tissues.

[0028] Furthermore, the systems and methods described herein can automatically optimize the operation of fluorescence imaging systems (e.g., fluorescence excitation illumination sources and / or fluorescence imaging devices) used during surgical procedures based on the identity of the fluorophore or the tissue type in which the fluorophore is present, thereby improving the quality of fluorescence images generated by the fluorescence imaging system. The systems and methods described herein can also automatically optimize the display of fluorescence images based on detected fluorescence based on the identity of the fluorophore or the tissue type in which the fluorophore is present. Each of these operations can improve the efficiency and effectiveness of surgical procedures. These and other benefits of the systems and methods described herein will become apparent in the following description.

[0029] Figure 1 A functional diagram of an exemplary imaging system 100 is shown, which can be used to capture visible light images and fluorescence images of a scene (e.g., a surgical area associated with a patient) according to the systems and methods described herein. As shown, the imaging system 100 includes an imaging device 102 and a controller 104. The imaging system 100 may include additional or alternative components, such as those that may serve a particular implementation. For example, the imaging system 100 may include various optical and / or electrical signal transmission components (e.g., wires, lenses, optical fibers, choke circuits, waveguides, etc.), cables housing wires and / or optical fibers and configured to interconnect the imaging device 102 and the controller 104, etc. While the imaging system 100 shown and described herein includes a fluorescence imaging system integrated with a visible light imaging system, the imaging system 100 may alternatively be implemented as a stand-alone fluorescence imaging system configured to capture only fluorescence images of a scene. Therefore, components of the imaging system 100 used only for capturing visible light (e.g., white light) images may be omitted. In some examples, a standalone fluorescence imaging system can be physically integrated with a visible light imaging system, such as by inserting the fluorescence imaging system into an auxiliary port of the endoscope.

[0030] like Figure 1 As shown, imaging device 102 can be used to capture visible light and fluorescence images of a scene. An exemplary scene includes patient tissue 106 and fluorophores 108 present within the tissue 106. The scene may also include... Figure 1Other objects not shown include surgical instruments. The fluorophore 108 can be any suitable fluorophore configured to emit fluorescence upon excitation by fluorescent illumination. Suitable fluorophores can include, for example, endogenous compounds (e.g., flavin adenine dinucleotide (FAD), reduced nicotinamide adenine dinucleotide (NADH), riboflavin, collagen, etc.) as well as exogenous compounds, organic dyes, proteins, quantum dots, organometallic complexes, lanthanides, fullerenes, nanotubes, etc. When used in the singular, fluorophore 108 refers to a specific type of fluorophore present in the scene (e.g., indocyanine green (ICG), fluorescein, rhodamine, etc.), whether as a single molecule or as a group of molecules. As will be explained in more detail, the imaging device 102 can capture visible light images of tissue 106 and / or other objects within the scene based on visible light 110 reflected by tissue 106 and other objects in the scene, and can capture fluorescence images based on fluorescence 112 emitted by fluorophore 108.

[0031] Imaging device 102 can be implemented by any suitable device configured to capture images of the scene. In some examples, such as Figure 1 As shown, the imaging device 102 is implemented using an endoscope. The imaging device 102 includes a camera 114, an axis 116 coupled to and extending away from the camera 114, an image sensor 118 (e.g., a visible light sensor 118-V and a fluorescence detection sensor 118-F), and an illumination channel 120. The imaging device 102 can be manually handled and controlled (e.g., by a surgeon performing a surgical procedure on a patient). Alternatively, the camera 114 can be coupled to a manipulator arm of a computer-assisted surgical system and controlled using robotics and / or teleoperation technologies. The distal end of the axis 116 can be positioned at or near the scene to be imaged by the imaging device 102. For example, the distal end of the axis 116 can be inserted into the patient.

[0032] The visible light sensor 118-V is configured to detect (e.g., capture, collect, sense, or otherwise acquire) visible light 110 reflected from tissue 106 and any objects (such as surgical instruments) included in the scene. As will be explained below, the visible light sensor 118-V can convert the detected visible light into data representing one or more visible light images.

[0033] The visible light sensor 118-V can be implemented by any suitable image sensor, such as a charge-coupled device (“CCD”) image sensor, a complementary metal-oxide-semiconductor (“CMOS”) image sensor, etc. In some examples, such as Figure 1As shown, the visible light sensor 118-V is positioned at the distal end of axis 116. Alternatively, the visible light sensor 118-V may be positioned closer to the proximal end of axis 116, inside camera 114, or outside imaging device 102 (e.g., inside controller 104). In these alternative configurations, optics (e.g., lenses, optical fibers, etc.) included in axis 116 and / or camera 114 can transmit light from the scene to the visible light sensor 118-V.

[0034] The fluorescence detection sensor 118-F is configured to detect (e.g., capture, collect, sense, or otherwise acquire) fluorescence 112 emitted by fluorophore 108. Fluorescence 112 may have wavelengths in the ultraviolet, visible, and / or infrared regions. As will be explained below, the fluorescence detection sensor 118-F can convert the detected fluorescence 112 into data representing one or more fluorescence images.

[0035] The fluorescence detection sensor 118-F can be implemented by any suitable sensor configured to detect fluorescence 112 and determine the lifetime of the detected fluorescence 112. As will be explained below, for example, the lifetime of the detected fluorescence can be determined according to time-domain techniques, frequency-domain techniques, or any other suitable techniques. Therefore, in some embodiments, the fluorescence detection sensor 118-F can be implemented by any suitable sensor configured for time-domain and / or frequency-domain determination of fluorescence lifetime. Suitable sensors include, but are not limited to, time-correlated single-photon counting (TCSPC) based photodetectors (e.g., single-photon counting detectors, photomultiplier tubes (PMTs), single-photon avalanche diode (SPAD) detectors, etc.), time-gated photodetectors (e.g., enhancement-mode CCDs), time-of-flight sensors, streak cameras, etc.

[0036] In an alternative embodiment, the fluorescence detection sensor 118-F may be implemented by multiple image sensors (such as CCD image sensors and / or CMOS image sensors), each configured to sample fluorescence at different time sequences. As will be explained below, such sensors typically sample too slowly individually to determine fluorescence lifetime (this typically occurs on the nanosecond scale). However, multiple such sensors can be uniquely configured to operate sequentially to collect sufficient fluorescence image signals to determine fluorescence lifetime. As an alternative to multiple differentiated image sensors, the fluorescence detection sensor 118-F may instead be implemented by a single sensor having multiple differentiated regions that sequentially sample fluorescence to collect sufficient fluorescence image signals to determine fluorescence lifetime. Exemplary multi-sensor and multi-region sensor configurations will be described in more detail below.

[0037] In some embodiments, the fluorescence detection sensor 118-F may be implemented by a plurality of differentiated sensors (or multiple regions on a single sensor), each sensor being tailored for a specific purpose (e.g., differentiated wavelengths of fluorescence). For example, a first fluorescence detection sensor may include a first filter configured to detect fluorescence in a first band (e.g., near-infrared), a second fluorescence detection sensor may include a second filter configured to detect fluorescence in a second band (e.g., ultraviolet), and a third fluorescence detection sensor may include a third filter configured to detect fluorescence in a third band (e.g., visible light). In additional or alternative examples, each sensor (or region of a sensor) may be configured to operate at different sampling rates, imaging parameters (e.g., exposure, gain, etc.).

[0038] The fluorescence detection sensor 118-F can be positioned at the distal end of axis 116, or alternatively, it can be positioned closer to the proximal end of axis 116, inside camera 114, or outside imaging device 102 (e.g., inside controller 104). In these alternative configurations, optics included in axis 116 and / or camera 114 can transmit fluorescence 112 from the scene to fluorescence detection sensor 118-F. In some examples, fluorescence detection sensor 118-F can share optics with visible light sensor 118-V.

[0039] The fluorescence detection sensor 118-F can capture images of all or part of the scene captured by the visible light sensor 118-V. In some examples, the field of view of the fluorescence detection sensor 118-F can be the same as that of the visible light sensor 118-V, but can be slightly different (due to its position within axis 116) without losing its effectiveness.

[0040] In some examples, the imaging device 102 is stereoscopic, in which case the visible light sensor 118-V includes two sensors configured to capture left and right visible images of the scene. Similarly, the fluorescence detection sensor 118-F may include two differential sensors configured to capture left and right fluorescence images of the scene. In other examples, the imaging device 102 is single-field-of-view, in which case the visible light sensor 118-V and / or the fluorescence detection sensor 118-F are configured to capture a single visible light image and a single fluorescence image, respectively.

[0041] Image sensor 118 can be configured to operate according to one or more definable (e.g., adjustable) parameters (e.g., activation time / sampling rate, exposure duration, auto exposure, gain, etc.).

[0042] The lighting channel 120 can be implemented by one or more optical components (e.g., optical fibers, light guides, lenses, etc.). As will be described below, the lighting channel 120 can be used to provide illumination to the scene to light it up.

[0043] The controller 104 can be implemented by any suitable combination of hardware and software configured to control the imaging device 102 and / or interface with the imaging device 102. For example, the controller 104 can be implemented at least in part by a computing device included in a computer-assisted surgical system.

[0044] Controller 104 includes a camera control unit (“CCU”) 122 and an illumination source 124 (e.g., a visible light illumination source 124-V and a fluorescence excitation illumination source 124-F). Controller 104 may include additional or alternative components, such as those servicing a particular implementation. For example, controller 104 may include circuitry configured to provide power to components included in imaging device 102. In some examples, CCU 122 and / or illumination source 124 are alternatively included in imaging device 102 (e.g., in camera 114).

[0045] CCU 122 can be configured to control (e.g., define, adjust, configure, set, etc.) any definable parameters of image sensor 118. CCU 122 can also be configured to receive and process image data from image sensor 118. Although CCU 122 is... Figure 1 While shown as a single unit, CCU 122 may instead be implemented by multiple CCUs, each configured to control distinct image streams (e.g., visible light image stream, fluorescence image stream, right-side fluorescence image stream, left-side fluorescence image stream, etc.).

[0046] Illumination source 124 can be configured to generate and emit illumination 126. Illumination 126 (also referred to herein as light) can travel through illumination channel 120 to the distal end of axis 116, where illumination 126 exits to illuminate the scene. Illumination 126 generated by visible light illumination source 124-V can include visible light 126-V having one or more color components or a continuous spectrum (e.g., white light). Illumination generated by fluorescence-excited illumination source 124-F can include fluorescence-excited illumination 126-F configured to excite fluorophore 108. Fluorescence-excited illumination 126-F can include one or more broadband spectra or can include one or more discrete wavelengths of light.

[0047] The illumination source 124 can be configured to operate according to one or more definable (e.g., adjustable) parameters (e.g., parameters specifying wavelength or band, waveform, intensity, frequency, pulse width, period, modulation, etc.). The illumination source 124 can be implemented by any suitable device, such as a flash lamp, laser source, laser diode, light-emitting diode, etc. Although each illumination source 124 is shown as a single device in the controller 104, each illumination source 124 may alternatively include multiple illumination sources, each configured to generate and emit illumination with different configurations. Alternatively, although the illumination source 124 is... Figure 1 The illumination source 124 is shown as multiple units, but it can be implemented as a single unit configured to emit both visible light 126-V and fluorescent excitation illumination 126-F.

[0048] To capture one or more images of a scene, controller 104 (or any other suitable computing device) may activate illumination source 124 and image sensor 118. When activated, illumination source 124 simultaneously emits illumination 126, which travels into the scene via illumination channel 120. Visible light sensor 118-V detects visible light 110 (e.g., portions of visible light 126-V reflected from one or more surfaces in the scene, such as tissue 106), and fluorescence detection sensor 118-F detects fluorescence 112 emitted by fluorophore 108 after being excited by fluorescence-excited illumination 126-F.

[0049] The visible light sensor 118-V (and / or other circuitry included in the imaging device 102) can convert the detected visible light 110 into visible light image data 128-V representing one or more visible light images of the scene. Similarly, the fluorescence detection sensor 118-F (and / or other circuitry included in the imaging device 102) can convert the detected fluorescence 112 into fluorescence image data 128-F representing one or more fluorescence images of the scene. The image data 128 (e.g., visible light image data 128-V and fluorescence image data 128-F) can have any suitable format.

[0050] Image data 128 is transmitted from image sensor 118 to CCU 122. Image data 128 can be transmitted via any suitable communication link between image sensor 118 and CCU 122. For example, image data 128 can be transmitted via a wire included in the cable connecting imaging device 102 and controller 104. Additionally or alternatively, image data 128 can be transmitted via one or more optical fibers.

[0051] CCU 122 can process (e.g., pack and / or format) image data 128 and output processed image data 130 (e.g., processed visible light image data 130-V corresponding to visible light image data 128-V and processed fluorescence image data 130-F corresponding to fluorescence image data 128-F). CCU 122 can transfer the processed image data 130 to an image processor (not shown) for further processing.

[0052] The image processor may be implemented by one or more computing devices external to the imaging system 100, such as one or more computing devices included in a computer-assisted surgical system. Alternatively, the image processor may be included in the controller 104. The image processor may prepare processed image data 130 for display by one or more display devices (e.g., in the form of one or more still images and / or video content). For example, the image processor may generate multiple visible light images based on the processed visible light image data 130-V, which may be sequentially output to form a visible light image stream. The visible light images may include panchromatic images and / or grayscale images. The image processor may also generate multiple fluorescence images based on the processed fluorescence image data 130-F, which may be sequentially output to form a fluorescence image stream. The system 100 may direct one or more display devices to then display the visible light image stream and / or fluorescence image stream.

[0053] In some examples, the image processor can combine (e.g., mix) processed visible light image data 130-V and processed fluorescence image data 130-F to generate multiple enhanced images, which can be sequentially output to form an enhanced image stream for display on one or more display devices. The enhanced images can display artificially colored (such as green or blue) fluorescent regions (derived from processed fluorescence image data 130-F) to highlight the fluorescent regions. Furthermore, the image processor can be configured to selectively apply gain to the fluorescence images to adjust (e.g., increase or decrease) the illumination intensity of the fluorescent regions. System 100 can direct one or more display devices to display the enhanced image stream.

[0054] In some examples, the image processor can operate based on one or more definable (e.g., adjustable) parameters. As will be explained in more detail below, the image processor can be configured to set the color of the fluorescent region based on the determined identity of fluorophore 108 or based on the type of tissue 106 in which fluorophore 108 is present, perform white balance, correct the processed image data 130, and perform other similar operations.

[0055] In some examples, the imaging system 100 is connected to, integrated into, or implemented by a surgical system. For instance, the imaging system 100 may be connected to, integrated into, or implemented by a computer-assisted surgical system that utilizes robotic and / or teleoperation technologies to perform surgical procedures (e.g., minimally invasive surgical procedures). Exemplary computer-assisted surgical systems are described herein.

[0056] In some cases, the imaging system 100 may not be aware of the identity of the fluorophore 108 present in tissue 106. Therefore, the operation of the imaging system 100 may not be optimally configured to capture fluorescence images, and the captured fluorescence images may not be optimally configured for display by a display device. In other cases, the identity of the fluorophore 108 present in tissue 106 may be known, but the type of tissue 106 in which the fluorophore 108 is present is unknown. Therefore, a fluorescence image based on fluorescence 112 may not convey useful information about the tissue 106 in which the fluorophore 108 is present.

[0057] To address these issues, the fluorescence imaging control system can be configured to determine the fluorescence lifetime of fluorescence 112. Based on the determined fluorescence lifetime, the fluorescence imaging control system can determine the identity of fluorophore 108 or the type of tissue 106 in which fluorophore 108 is present. The fluorescence imaging control system can also be configured to control the operation of imaging system 100 and / or image processor based on the determined identity of fluorophore 108 or the type of tissue 106 in which fluorophore 108 is present.

[0058] Figure 2 An exemplary fluorescence imaging control system 200 (“System 200”) is illustrated, which can be configured to determine the identity of fluorophores present at a scene or the type of tissue in which fluorophores are present and / or identify distinguishable regions in which different tissue types contain fluorophores. System 200 can be included in, implemented by, or connected to any surgical system or other computing system described herein. For example, System 200 can be implemented by a computer-assisted surgical system. As another example, System 200 can be implemented by a separate computing system communicatively coupled to a computer-assisted surgical system.

[0059] As shown in the figure, system 200 includes, but is not limited to, storage facility 202 and processing facility 204 that are selectively and communicatively coupled to each other. Facilities 202 and 204 may each include or be implemented by hardware and / or software components (e.g., processor, memory, communication interface, instructions stored in memory for execution by the processor, etc.). For example, facilities 202 and 204 may be implemented by any component of a computer-assisted surgical system. In some examples, facilities 202 and 204 may be distributed among multiple devices and / or multiple locations, as may be necessary for a particular implementation.

[0060] Storage facility 202 may maintain (e.g., store) executable data used by processing facility 204 to perform any of the operations described herein. For example, storage facility 202 may store instructions 206 that can be executed by processing facility 204 to perform any of the operations described herein. Instructions 206 may be implemented by any suitable application, software, code, and / or other instance of executable data. Storage facility 202 may also maintain any data received, generated, managed, used, and / or transmitted by processing facility 204.

[0061] Processing facility 204 can be configured to perform (e.g., execute instructions 206 stored in storage facility 202) various operations associated with determining the identity of a fluorophore present in a scene or identifying the type of tissue in which the fluorophore is present or a distinguishable region of a different tissue type. For example, processing facility 204 can be configured to: guide an illumination source to illuminate a scene with fluorescence excitation illumination configured to excite a fluorophore present in the scene; and guide an imaging device to detect fluorescence emitted by the fluorophore in response to excitation by the fluorescence excitation illumination. Processing facility 204 can also be configured to determine the lifetime of fluorescence based on the detected fluorescence. Processing facility 204 can be configured to determine the identity of a fluorophore or identify the type of tissue in which the fluorophore is present or a distinguishable region based on the determined lifetime of fluorescence. In some examples, processing facility 204 can be configured to perform one or more operations based on the determined identity of the fluorophore or based on the determined type of tissue in which the fluorophore is present. These and other operations that can be performed by processing facility 204 are described herein. In the following description, any reference to the operations performed by system 200 can be understood as being performed by the processing facility 204 of system 200.

[0062] As described above, system 200 can be configured to determine the lifetime of fluorescence emitted by fluorophores present in the scene. The lifetime can be measured or determined in any suitable manner, including but not limited to time-domain or frequency-domain methods.

[0063] In the temporal method, system 200 is configured to guide an illumination source (e.g., a fluorescence excitation illumination source 124-F) to illuminate a scene with short pulses of fluorescence excitation illumination (e.g., fluorescence excitation illumination 126-F), which is configured to excite fluorophores present at the scene (e.g., fluorophore 108).

[0064] Figure 3 A curve 302 representing an exemplary pulsed fluorescence excitation illumination is shown, with time shown on the x-axis and intensity shown on the y-axis. As shown in curve 302, the fluorescence excitation illumination has multiple pulses 304. System 200 can set the pulse width and period (or frequency) to any value, as may be suitable for a particular implementation. In some examples, the period is greater than or equal to the lifetime of fluorescence emitted by the fluorophore excited by the fluorescence excitation illumination, and the pulse width is shorter than the expected fluorescence lifetime. In some examples, the pulse width is less than about 100 nanoseconds (ns). In other examples, the pulse width is less than about 10 ns. In yet another example, the pulse width is less than about 3 ns.

[0065] System 200 can control the fluorescent excitation illumination source in any suitable manner. Figure 3 An exemplary drive signal 306 is shown, configured to drive the operation of a fluorescent excitation illumination source to generate fluorescent excitation illumination, represented by curve 302. As shown, the drive signal 306 is a rectangular pulse wave having a plurality of pulses 308 configured to generate pulses 304 of fluorescent excitation illumination. In some examples, the drive signal 306 electrically controls the on / off state of the fluorescent excitation illumination source. Alternatively, the drive signal 306 mechanically controls the output of the fluorescent excitation illumination, such as by setting the speed of a rotating wheel that periodically allows the transmission of fluorescent excitation illumination.

[0066] System 200 is also configured to guide an imaging device (e.g., a fluorescence detection sensor 118-F) to detect fluorescence emitted by a fluorophore in response to excitation of the fluorophore by fluorescence excitation illumination (e.g., excitation of the fluorophore by a pulse 304 of fluorescence excitation illumination). The imaging device measures the intensity of the emitted fluorescence over time, including after the termination of fluorescence excitation illumination (e.g., after each pulse 304). Any suitable method can be used to detect fluorescence, including but not limited to TCSPC and fast time gating (e.g., time gating with multiple equal-width gates, time-gated scanning, etc.). In some examples, the fluorescence detection sensor is implemented by a PMT or avalanche photodiode (e.g., a SPAD) configured to record the temporal correlation distribution of emitted photons at each location (e.g., at each pixel) after one or more pulses 304 of fluorescence excitation illumination. In alternative examples, the fluorescence detection sensor is implemented by a fast time-gated image intensifier (e.g., an intensified CCD) configured to measure fluorescence intensity over a series of different time windows.

[0067] Figure 3 An exemplary activation timing signal 310 is shown for activating an imaging device according to a time-gated method to detect fluorescence excitation illumination. As shown, the activation timing signal 310 is configured to drive the imaging device to detect fluorescence excitation illumination at multiple differentiated times after the termination of fluorescence excitation illumination (e.g., after each pulse 304 of fluorescence excitation illumination), thereby collecting multiple fluorescence image signal data points representing intensity as a function of time. As will be explained in more detail later, these data points can then be used to generate a decay profile from which fluorescence lifetime can be determined.

[0068] In some examples, the imaging device may be implemented using a multi-sensor and / or multi-region sensor configuration. Figure 4A An exemplary multi-region sensor 400 is shown, configured to detect fluorescence to determine its lifetime. Sensor 400 can be implemented by any suitable sensor, such as a CCD or CMOS sensor. Sensors such as CCDs or CMOS typically cannot sample light fast enough to determine the lifetime of detected fluorescence. To address this issue, sensor 400 is divided into multiple regions 402, which are configured to sequentially sample fluorescence. Figure 4A As shown, sensor 400 is divided into four regions (i.e., regions 402-1 to 402-4), but sensor 400 can be divided into more or fewer regions 402, as may be suitable for a particular implementation. Each region 402 consists of multiple pixels and is configured to capture a fluorescence image of the scene. In a multi-sensor configuration, regions 402 can be implemented using distinct, independent sensors.

[0069] Each region 402 is configured to be activated sequentially at different times to generate multiple fluorescence images, which can then be combined to generate a decay curve from which the fluorescence lifetime can be determined, as described in more detail below. Figure 4B An exemplary timing sequence for the activation of region 402 is shown. At time t1, region 402-1 is activated to detect fluorescence until region 402-1 is deactivated. Region 402-2 is activated at time t2, region 402-3 is activated at time t3, and region 402-4 is activated at time t4. In this way, sensor 400 is configured to generate multiple fluorescence image signals, wherein each region 402 generates a distinct fluorescence image signal. The timing between the activations of different regions 402 is set fast enough that sensor 400 is configured to collect multiple fluorescence image signals during the fluorescence lifetime of the detected fluorescence. Although the fluorescence lifetime is initially unknown, the timing of the activation of region 402 can be estimated or initially set based on a predetermined value and can be iteratively adjusted until sufficient data is acquired.

[0070] As described above, system 200 can be based on the fluorescence generation decay curve detected by the imaging device. The decay curve can be generated by any suitable method, depending on the detection method used (e.g., TCSPC or time-gated). See again... Figure 3 An exemplary decay curve 312 is shown. Decay curve 312 is generated based on the detected fluorescence and plots the intensity of the detected fluorescence as a function of time. It can be seen that the intensity of the emitted fluorescence reaches a peak in response to the pulse 304 of fluorescence excitation illumination and then decays over time according to the above equation [1]. The fluorescence lifetime τ occurs when the intensity value decays to 1 / e (~36.8%).

[0071] System 200 can be configured to determine the fluorescence lifetime τ from the slope of decay curve 312 according to equation [1]. The fluorescence lifetime τ can be calculated using any suitable curve fitting algorithm, such as the least squares curve method. If the pulse width of the fluorescence excitation illumination is comparable to the fluorescence lifetime τ, then the deconvolution method can be used additionally or alternatively to determine the fluorescence lifetime τ.

[0072] As described above, system 200 can alternatively determine the lifetime of the detected fluorescence using a frequency domain method. In this method, system 200 is configured to guide a fluorescence excitation source to illuminate a scene by sinusoidally modulating the fluorescence excitation illumination. As a result, the fluorescence emitted by the fluorophore is also modulated at the same frequency as the fluorescence excitation illumination, but with a phase shift and modulation depth (change in amplitude) relative to the fluorescence excitation illumination.

[0073] System 200 is configured to guide an imaging device to detect sinusoidally modulated fluorescence emitted by a fluorophore in response to excitation of the fluorophore by fluorescence excitation illumination. Fluorescence can be detected using any sensor suitable for frequency domain detection, such as, but not limited to, a PMT or CCD (e.g., an intensified CCD) with a gain modulator. Phase shift and modulation depth can be detected from the detected fluorescence using any suitable method, such as a zero-difference detection method. For example, the signal detected by the sensor is modulated (or gated) at the same frequency as the fluorescence excitation illumination. However, the phase of the image intensifier is shifted relative to the fluorescence excitation illumination over a series of fixed steps over time, thereby generating a low-pass signal for each pixel. The phase shift and modulation depth can then be obtained from this low-pass frequency domain signal.

[0074] System 200 is configured to determine fluorescence lifetime based on detected fluorescence. For example, according to equations [2] and [3], the detected phase shift and modulation depth are related to fluorescence lifetime and modulation frequency of fluorescence excitation illumination:

[0075]

[0076]

[0077] Where ω is the modulation frequency of the fluorescence excitation illumination, and τ is the fluorescence lifetime of the detected fluorescence. M is the detected phase shift, and M is the detected modulation depth. Therefore, the phase shift can be measured according to equation [2] and / or equation [3]. The fluorescence lifetime τ is determined by the modulation depth M and / or the modulation depth M. Any suitable curve fitting algorithm can be used to determine the fluorescence lifetime, such as the least squares curve method.

[0078] The foregoing examples have described determining the fluorescence lifetime of detected fluorescence-excited illumination using time-domain or frequency-domain methods. However, system 200 is not limited to these methods, and any other suitable method can be used to determine the lifetime of fluorescence detected by the imaging device. Furthermore, the foregoing examples have described illuminating a scene with pulsed fluorescence-excited illumination. However, the fluorescence-excited illumination does not need to be pulsed, but can have any other configuration, as may be suitable for a particular implementation. In some configurations where the fluorescence-excited illumination is continuous, the fluorescence lifetime can be determined by pausing the illumination of the scene when the imaging device detects fluorescence decay. Illumination can be paused any number of times and at any time (e.g., at regular intervals, randomly, in response to a specific event, etc.), as may be suitable for a particular implementation.

[0079] As described above, in situations where the identity of a fluorophore present in a scene is unknown, system 200 can be configured to determine the identity of the fluorophore present in the scene based on the determined lifetime of the detected fluorescence (whether determined by a time-domain method, a frequency-domain method, or any other method). System 200 can determine the identity of the fluorophore in any suitable manner.

[0080] In some examples, system 200 can determine the identity of a fluorophore by accessing a fluorophore lookup table, which includes fluorophore data representing the fluorescence lifetime of fluorescence emitted by each of a plurality of fluorophores. For example, the fluorophore lookup table may list a plurality of fluorophores and the fluorescence lifetime of each fluorophore. In some examples, the fluorescence lifetime of each fluorophore may be derived empirically. System 200 can select a specific fluorophore from the plurality of fluorophores that has a fluorescence lifetime that best matches the determined lifetime of fluorescence emitted by fluorophores present in the scene. For illustration, system 200 may determine that the fluorescence lifetime of detected fluorescence 112 is 4.0 ns. Therefore, system 200 can access the fluorophore lookup table and determine that the fluorescence lifetime of fluorophore 108 best matches the fluorescence lifetime of luciferin, since luciferin also has a fluorescence lifetime of 4.0 ns.

[0081] In some examples, the fluorophore data included in the fluorophore lookup table may also include information on other fluorophore properties to aid in fluorophore identification. Such information may include, but is not limited to, optical properties such as the wavelength indicating peak fluorescence excitation, peak fluorescence emission, fluorescence intensity under different excitation sources, peak fluorescence wavelength and peak offset in various solvents or biological fluids (e.g., solutions where the fluorophore is present for a given fluorescence lifetime, such as lymph, blood, etc.); tissue type (e.g., the type of tissue where the fluorophore is present for a given fluorescence lifetime and / or the disease state of the tissue), etc. If the fluorophore lookup table includes multiple different fluorophores with the same fluorescence lifetime, the identity of the fluorophores present at the scene can be identified based on one or more additional properties. For example, the wavelength of the fluorescence excitation illumination when the imaging device detects fluorescence can be used to select a specific fluorophore from the multiple fluorophores in the fluorophore lookup table that has both a matching fluorescence lifetime and a matching peak fluorescence excitation wavelength.

[0082] In some examples, there may be more than one fluorophore at a scene. For example, multiple fluorophores may be applied to a patient, each configured to convey different information. The principles described herein allow system 200 to identify the multiple fluorophores present at a scene. For example, system 200 may be configured to guide an illumination source to illuminate the scene with fluorescence excitation illumination configured to excite a first fluorophore and a second fluorophore present at the scene. For example, the fluorescence excitation illumination may include illumination of varying wavelengths (e.g., in the infrared band (e.g., ranging from about 760 nm to about 1000 nm), the ultraviolet band (e.g., ranging from about 280 nm to about 380 nm), or the visible band (e.g., ranging from about 380 nm to about 760 nm)). In this way, the fluorescence excitation illumination may be configured to excite both the first and second fluorophores, even if they have different peak fluorescence excitation illumination wavelengths. Alternatively, system 200 may be configured to guide one or more fluorescence excitation illumination sources to illuminate the scene with multiple uniquely configured pulse waves of fluorescence excitation illumination, each pulse wave configured to excite a specific fluorophore.

[0083] System 200 can also guide the imaging device to detect a first fluorescence emitted by the first fluorophore and a second fluorescence emitted by the second fluorophore in response to the excitation of a first fluorophore and a second fluorophore by fluorescence excitation illumination. For example, the first fluorophore and the second fluorophore may be located at different locations in the scene. Therefore, the imaging device can detect the first fluorescence emitted by the first fluorophore in a first pixel or pixel region of the imaging device, and can detect the second fluorescence emitted by the second fluorophore in a second pixel or pixel region of the imaging device. When the first fluorophore and the second fluorophore are located at the same location such that the emitted fluorescence overlaps (e.g., the pixel detects a mixture of the first and second fluorescence), the first and second fluorescence can be separated in any suitable manner. In some examples, the first and second fluorescence are spatially separated by optical components (e.g., dichroic filters, mirrors, lenses, etc.) configured to guide the first and second fluorescence onto a differentiated region of a fluorescence detection sensor (or a differentiated fluorescence detection sensor). As another example, the first and second fluorescence are temporally separated by sequentially pulsed multiple pulses of fluorescence excitation illumination and sequentially detecting the first and second fluorescence in sync with the pulses of fluorescence excitation illumination.

[0084] System 200 can then determine the lifetime of the first fluorescence based on the detected first fluorescence, and can determine the lifetime of the second fluorescence based on the detected second fluorescence. The lifetimes of the first and second fluorescence can be determined in any of the ways described herein.

[0085] System 200 can also identify the first fluorophore based on the determined lifetime of the first fluorescence, and can identify the second fluorophore based on the determined lifetime of the second fluorescence.

[0086] While the foregoing examples have described the identification of a first and second fluorophore present in a scene, system 200 can be configured to identify any number of fluorophores present in a scene. As can be seen from the preceding description, the systems and methods described herein are capable of identifying multiple fluorophores present in a scene using a single sensor and / or a single fluorescence excitation illumination source. Furthermore, the systems and methods described herein do not require the use of wavelength filters to distinguish fluorescence emitted from multiple different fluorophores present in a scene.

[0087] As described above, system 200 can be configured to control the operation of one or more components of the fluorescence imaging system (e.g., image sensor 118 and / or illumination source 124 of imaging system 100) and / or image processor based on the determined identity of the fluorophore.

[0088] For example, system 200 can be configured to set the fluorescence excitation illumination (e.g., fluorescence excitation illumination 126-F) based on the optical properties of the identified fluorophore. The configuration of the fluorescence excitation illumination refers, for example, to the wavelength or band, waveform, intensity, frequency, pulse width, period, and modulation of the fluorescence excitation illumination. System 200 can configure the fluorescence excitation illumination in any suitable manner. For example, system 200 can adjust the operating settings of the fluorescence excitation illumination source (e.g., fluorescence excitation illumination source 124-F) and / or the mechanical components of the fluorescence excitation illumination source (e.g., filters, lenses, rotating wheels, etc.). System 200 can configure the fluorescence excitation illumination based on the determined identity of the fluorophore, for example, thereby optimizing the emitted fluorescence for detection by an imaging device, improving the quality of the signal detected by the imaging device, increasing the quantity and accuracy of data acquired for determining fluorescence lifetime, saving battery power, reducing processing power, preventing the emission of certain fluorescence, and so on.

[0089] To illustrate, system 200 can configure the wavelength or band of fluorescence excitation illumination to include and / or focus on the peak fluorescence excitation illumination wavelength of fluorophores present in the scene, thereby optimizing the fluorescence intensity for detection by an imaging device. For example, if system 200 determines that fluorophore 108 is ICG, which has a peak fluorescence excitation illumination wavelength of approximately 780 nm in water, system 200 can adjust fluorescence excitation illumination 126-F to include illumination with a wavelength of approximately 780 nm. In some examples, system 200 can also narrow the band of fluorescence excitation illumination 126-F to emit only the peak fluorescence excitation illumination necessary to excite fluorophore 108. In alternative examples, such as when the fluorescence image is irrelevant or unwanted by the user, system 200 can adjust fluorescence excitation illumination 126-F to exclude illumination configured to excite fluorophore 108 (e.g., illumination with a wavelength of approximately 780 nm).

[0090] As another illustration, system 200 can configure the frequency of fluorescence excitation illumination 126-F based on the lifetime of fluorescence 112 emitted by fluorophore 108. For example, if system 200 determines that fluorophore 108 is coumarin 6 and has a lifetime of 2.5 ns, system 200 can adjust the frequency of fluorescence excitation illumination 126-F. This can increase the average intensity (brightness) of the fluorescence image signal, thereby improving the quality of the fluorescence image. As another example, if system 200 determines that fluorophore 108 is pyrene and has a lifetime of 70 ns or longer, system 200 can reduce the frequency of excitation illumination 126-F to reduce the number of pulses 304 (see [link to documentation]). Figure 3 This ensures that a sufficient number of data points are collected for fluorescence lifetime determination, thereby improving the accuracy of fluorescence lifetime determination.

[0091] As described above, the image processor can be configured to generate an image (e.g., a fluorescence image or an enhanced image) based on the detected fluorescence for presentation by a display device. System 200 can also be configured to process captured images (e.g., processed image 130) for presentation by the display device based on the determined identity, distribution, and / or concentration of the fluorophore. For example, system 200 can specify the color(s) of the fluorescent region(s), the intensity (brightness) of the fluorescent region, the method of mixing the fluorescence image and the visible light image, image correction, white balance adjustment, etc. To illustrate, if system 200 determines that fluorophore 108 is ICG, system 200 can instruct the image processor to pseudo-color the fluorescent region to a specific color (e.g., green) and increase the intensity (brightness) of the fluorescent region in the fluorescence image. On the other hand, if system 200 determines that fluorophore 108 is fluorescein, system 200 can instruct the image processor to pseudo-color the fluorescent region to a different color (e.g., blue) and decrease the intensity of the fluorescent region.

[0092] System 200 can also be configured to configure an imaging device (e.g., fluorescence detection sensor 118-F) based on the determined identity of the fluorophore (e.g., based on the optical properties of the fluorophore). The configuration of the imaging device refers, for example, to the sampling rate of the imaging device, the activation period of the imaging device (e.g., exposure time), the gain applied to the detected fluorescence signal, the activation timing of the imaging device, etc. In some examples where the imaging device is implemented by multiple sensors (or by multiple distinct regions on the sensors), the configuration of the imaging device may additionally or alternatively refer to the on / off state of each sensor (or each region of the sensor). System 200 can configure the imaging device in any suitable manner. For example, system 200 can adjust the operating settings of the imaging device and / or adjust the mechanical components of the imaging device (e.g., filters, shutter, etc.). System 200 can configure the imaging device based on the determined identity of the fluorophore, for example, to optimize the quality of the signal detected by the imaging device, increase the quantity and accuracy of data acquired for determining fluorescence lifetime, save battery power, reduce processing power, exclude the detection of certain fluorescence signals, etc.

[0093] To illustrate, if system 200 determines that fluorophore 108 is of a type with a fluorescence lifetime greater than 10 ns, system 200 can instruct the imaging device to reduce the sampling rate. As another illustration, if system 200 determines that fluorophore 108 is of a type with a peak fluorescence wavelength of 803 nm, system 200 can instruct the imaging device to shut down any sensors not configured to detect emitted fluorescence (e.g., sensors configured to detect ultraviolet and / or visible light) (or discard signals detected by these sensors).

[0094] In embodiments where system 200 determines the identities of multiple distinct fluorophores present at a scene, system 200 can be configured to control the operation of one or more components of an imaging system and / or an image processor based on the determined identity of each of the multiple fluorophores present at the scene. System 200 can control the operation of the fluorescence imaging system components and / or the image processor in any of the above-described manner.

[0095] For example, if system 200 determines the identity of a first fluorophore present in the scene and the identity of a second fluorophore present in the scene, as described above, system 200 can guide the image processor to pseudo-color the fluorescent region with different colors based on the determined identities of the first and second fluorophores. Figure 5An exemplary enhanced image 500 is shown, in which fluorescent regions corresponding to different fluorophores are pseudo-colored differently. As shown, the fluorescence image 500 includes fluorescent regions 502 (e.g., fluorescent regions 502-1 to 502-5). Fluorescent regions 502-1, 502-3, and 502-5 are generated based on fluorescence emitted by a first fluorophore (e.g., ICG), and fluorescent regions 502-2 and 502-4 are generated based on fluorescence emitted by a second fluorophore (e.g., fluorescein). Therefore, fluorescent regions 502-1, 502-3, and 502-5 are pseudo-colored with a first color (e.g., green), while fluorescent regions 502-2 and 502-4 are pseudo-colored with a second color (e.g., blue). In this way, a person viewing the fluorescence image 500 can easily identify and distinguish the fluorescent regions corresponding to different fluorophores.

[0096] As described above, in some cases, the identity of a fluorophore present in a tissue at a scene may be known, but the type of tissue or the boundaries between regions of different tissue types in which the fluorophore is present may be unknown. Fluorescence images generated from fluorescence emitted from a fluorophore under continuous fluorescence excitation illumination may not convey useful information about the tissue in which the fluorophore is present, and fluorescence intensity in standard fluorescence imaging techniques exhibits limited variation depending on the optical properties of the tissue environment. However, the fluorescence lifetime of fluorescence emitted by certain fluorophores can vary (in some cases significantly) depending on the type of tissue in which the fluorophore is present. Changes in fluorescence lifetime (especially large changes) of a particular tissue type relative to adjacent tissue regions can clearly delineate the tissue of interest. Therefore, system 200 can be configured to use this characteristic to identify one or more distinct regions of different tissue types in which fluorophores are present based on the determined lifetime of the detected fluorescence.

[0097] As used in this article, tissue type can refer to tissues with distinct categories, such as connective tissue, muscle tissue, nervous tissue, epithelial tissue, and / or any other category of tissue, such as bone tissue, tissues of specific organs (e.g., liver tissue, lung tissue, etc.). Tissue type can also refer to the condition or state of a tissue, such as healthy / normal, diseased, cancerous, infected, bruised, swollen, etc.

[0098] System 200 can be configured to identify one or more distinct tissue regions in any suitable manner. In some embodiments, system 200 can identify one or more distinct tissue regions by pseudo-coloring pixels of a fluorescence image based on a determined lifetime of detected fluorescence at each pixel. For example, system 200 can generate a fluorescence image comprising multiple pixels based on detected fluorescence emitted by fluorophores present in multiple tissue types at a scene. System 200 can generate the fluorescence image by determining the lifetime of detected fluorescence at each of the multiple pixels based on the detected fluorescence and pseudo-coloring the multiple pixels based on the lifetime of detected fluorescence at each of the multiple pixels. Regions of similar tissue types will therefore be pseudo-colored to similar colors.

[0099] Figure 6A An exemplary method for pseudo-coloring multiple pixels included in a fluorescence image is illustrated. As shown, a fluorescence image 600A includes fluorescent regions 602 (e.g., fluorescent regions 602-1 to 602-5). Fluorescent regions 602 are based on fluorescence emitted by fluorophores of the same type (e.g., ICG) present in the scene. To pseudo-colorize the fluorescent regions 602, system 200 may apply a color scale 604, which assigns a color (or a shade of color in a monochrome image) to a fluorescence lifetime value. In some examples, the color scale 604 is pre-configured based on known fluorescence lifetime values ​​of fluorophores for use with specific fluorophores. For illustration, fluorophores known to be present in surgical areas associated with patients emit fluorescence from various tissue types with fluorescence lifetimes ranging from about 0 ns to about 10 ns. Therefore, the color scale for a fluorophore can assign a range of visible colors ranging from, for example, purple (~0 ns) to yellow (~5 ns) to red (~10 ns). Pixels where no fluorescence is detected can be pseudo-colored, for example, as black or white. As an alternative to pre-configured color scales, system 200 can set color scale 604 based on the range of fluorescence lifetimes determined from the detected fluorescence.

[0100] like Figure 6AAs shown, the pixels in fluorescent region 602 are pseudo-colored based on the fluorescence lifetime of the detected fluorescence at each pixel. For example, the fluorescence lifetime of the detected fluorescence at pixels in fluorescent regions 602-1, 602-4, and 602-5 is approximately 9 ns, therefore the pixels in fluorescent regions 602-1, 602-4, and 602-5 are pseudo-colored with a color corresponding to 9 ns (e.g., red), as specified by color bar 604. The fluorescence lifetime of the detected fluorescence at pixels in fluorescent region 602-2 is approximately 5 ns, therefore the pixels in fluorescent region 602-2 are pseudo-colored with a color corresponding to 5 ns (e.g., yellow), as specified by color bar 604. The fluorescence lifetime of the detected fluorescence at pixels in fluorescent region 602-3 is approximately 3 ns, therefore the pixels in fluorescent region 602-3 are pseudo-colored with a color corresponding to 3 ns (e.g., green), as specified by color bar 604.

[0101] In other examples, system 200 may pseudo-color each pixel of the fluorescence image based on changes in measured fluorescence lifetime values ​​of one or more neighboring pixels. For example, system 200 may apply a color scale that assigns a specific color (or shade of color) to a specific difference. In some examples, the difference may be the fluorescence lifetime of a first pixel and the fluorescence lifetime of a second pixel adjacent to the first pixel in a specific direction (e.g., horizontal from left to right, vertical from top to bottom, etc.). In other examples, the difference may be calculated based on multiple neighboring pixels.

[0102] By pseudo-coloring multiple pixels in a fluorescence image based on the determined lifetime of fluorescence detected at each of the multiple pixels, a fluorescence image (or an enhanced image based on a fluorescence image) can clearly identify regions of different tissue types. For example, a surgeon viewing image 600A can easily identify regions of cancerous tissue (e.g., fluorescent region 602-3) and regions of healthy tissue (e.g., fluorescent regions 602-1, 602-2, 602-4, and 602-5).

[0103] In some embodiments, system 200 may be additionally or alternatively configured to identify tissue types in which fluorophores are present. In some examples, system 200 may identify tissue types by accessing a tissue lookup table that includes tissue data representing the fluorescence lifetime of fluorescence emitted by a particular fluorophore in each of a plurality of distinct tissue types. The tissue lookup table may indicate that fluorescence emitted by a particular fluorophore has a specific lifetime (e.g., 2.3 ns) in healthy lung tissue and another specific lifetime (e.g., 2.7 ns) in cancerous lung tissue. System 200 may then select a particular tissue type from the plurality of distinct tissue types in which the fluorophore has a fluorescence lifetime that best matches the determined lifetime of the detected fluorescence. For illustration, system 200 may determine that the fluorescence lifetime of detected fluorescence 112 emitted by fluorophore 108 is 4.0 ns. System 200 can then access a tissue lookup table and determine that the fluorescence 112 emitted by fluorophore 108 is 3.2 ns in healthy lung tissue and 4.0 ns in cancerous lung tissue. As a result, system 200 can determine that the tissue 106 in which fluorophore 108 is present is cancerous lung tissue based on the determined lifetime of fluorescence 112. In some examples, the fluorophore lookup table and the tissue lookup table are implemented using a single lookup table configured for fluorophore and tissue type determination.

[0104] System 200 can be configured to control the operation of one or more components of an imaging system (e.g., image sensor 118 and / or illumination source 124 of imaging system 100) and / or image processor based on a determined tissue type. System 200 can be configured to control the operation of components of a fluorescence imaging system and / or image processor in any of the above-described manner.

[0105] In some examples, system 200 can instruct the image processor to pseudo-color fluorescent regions in a fluorescence image with different colors based on the determined tissue type. Figure 6B An exemplary method for pseudo-staining fluorescent regions based on the determined tissue type is shown. Figure 6B Similar to Figure 6AIn addition to the pseudo-coloring of fluorescent regions 602 based on the identified tissue type for each fluorescent region, fluorescent regions 602-1, 602-3, and 602-5 are generated based on fluorescence emitted by fluorophores from a first tissue type (e.g., healthy lung tissue), while fluorescent regions 602-2 and 602-4 are generated based on fluorescence emitted by fluorophores from a second tissue type (e.g., cancerous lung tissue). Therefore, fluorescent regions 602-1, 602-3, and 602-5 are pseudo-colored with a first color (e.g., green), while fluorescent regions 602-2 and 602-4 are pseudo-colored with a second color (e.g., red). In this way, a person viewing the fluorescence image 600 can easily identify and distinguish different types of tissue (e.g., distinguish between cancerous and healthy tissue) using a single fluorophore.

[0106] Figure 7 An exemplary computer-assisted surgical system 700 (“surgical system 700”) is illustrated. As described herein, system 200 may be implemented by, connected to, and / or otherwise used in conjunction with surgical system 700.

[0107] As shown in the figure, the surgical system 700 may include a control system 702, a user control system 704, and an auxiliary system 706 that are communicatively coupled. The surgical system 700 can be used by a surgical team to perform computer-assisted surgical procedures on a patient 708. As shown, the surgical team may include a surgeon 710-1, an assistant 710-2, a nurse 710-3, and an anesthesiologist 710-4, all of whom can be collectively referred to as "surgical team members 710". Additional or replacement surgical team members may appear during a surgical session, as may be necessary for a particular implementation.

[0108] Although Figure 7 A minimally invasive surgical procedure in progress is illustrated; however, it should be understood that the surgical system 700 can be similarly used to perform open surgical procedures or other types of surgical procedures that can similarly benefit from the accuracy and convenience of the surgical system 700. Furthermore, it should be understood that surgical sessions that can utilize the surgical system 700 from beginning to end may include not only procedures such as… Figure 7 The surgical phases of the surgical procedure illustrated may also include preoperative, postoperative, and / or other appropriate phases of the surgical procedure. A surgical procedure may include any procedure involving the use of manual and / or instrumental techniques to investigate or treat the patient's physical condition.

[0109] like Figure 7As shown, the manipulation system 702 may include a plurality of manipulator arms 712 (e.g., manipulator arms 712-1 to 712-4), to which a plurality of surgical instruments may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool with tissue interaction capabilities), medical instrument, imaging device (e.g., an endoscope), sensing device (e.g., a force-sensing surgical instrument), diagnostic instrument, or similar instrument that may be used to perform computer-assisted surgical procedures on a patient 708 (e.g., by at least partially inserting into and being manipulated to perform computer-assisted surgical procedures on the patient 708). While the manipulation system 702 is depicted and described herein as comprising four manipulator arms 712, it will be appreciated that the manipulation system 702 may include only a single manipulator arm 712 or any other number of manipulator arms, as may be served in a particular implementation.

[0110] The manipulator arm 712 and / or the surgical instruments attached to the manipulator arm 712 may include one or more displacement transducers, orientation sensors, and / or position sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of the surgical system 700 may be configured to use kinematic information to track (e.g., determine their position and orientation) and / or control the surgical instruments.

[0111] User control system 704 can be configured to facilitate surgeon 710-1's control of manipulator arm 712 and surgical instruments attached to manipulator arm 712. For example, surgeon 710-1 can interact with user control system 704 to remotely move or manipulate manipulator arm 712 and surgical instruments. To this end, user control system 704 can provide surgeon 710-1 with images (e.g., high-resolution 3D images, synthetic medical images, etc.) of the surgical area associated with patient 708 captured by an imaging system (e.g., imaging system 100). In some examples, user control system 704 may include a stereoscopic viewer with two displays, where surgeon 710-1 can view stereoscopic images of the surgical area associated with patient 708 and generated by a stereoscopic imaging system. Surgeon 710-1 can use the images to perform one or more procedures with one or more surgical instruments attached to manipulator arm 712.

[0112] To facilitate control of surgical instruments, the user control system 704 may include a set of master controls. These master controls can be manipulated by the surgeon 710-1 to control the movement of surgical instruments (e.g., by utilizing robotic and / or teleoperation technologies). The master controls can be configured to detect various hand, wrist, and finger movements of the surgeon 710-1. In this way, the surgeon 710-1 can intuitively perform procedures using one or more surgical instruments.

[0113] The auxiliary system 706 may include one or more computing devices configured to perform the main processing operations of the surgical system 700. In such a configuration, the one or more computing devices included in the auxiliary system 706 may control and / or coordinate operations performed by various other components of the surgical system 700 (e.g., the manipulation system 702 and the user control system 704). For example, the computing device included in the user control system 704 may transmit instructions to the manipulation system 702 via the one or more computing devices included in the auxiliary system 706. As another example, the auxiliary system 706 may receive and process image data representing images captured by an imaging device attached to one of the manipulator arms 712 from the manipulation system 702.

[0114] In some examples, the assistive system 706 may be configured to present visual content to surgical team members 710 who may not have access to the images provided to surgeon 710-1 at the user control system 704. For this purpose, the assistive system 706 may include a display monitor 714 configured to display one or more user interfaces, such as images of the surgical area (e.g., 2D images, synthetic medical images, etc.), information associated with patient 708 and / or surgical procedures, and / or any other visual content that may be available for a particular implementation. For example, the display monitor 714 may display images of the surgical area along with additional content displayed concurrently with the images (e.g., graphical content, contextual information, etc.). In some embodiments, the display monitor 714 is implemented by a touchscreen display that surgical team members 710 can interact with (e.g., via touch gestures) to provide user input to the surgical system 700.

[0115] The operating system 702, the user control system 704, and the auxiliary system 706 can be communicatively coupled to each other in any suitable manner. For example, Figure 7 As shown, the operating system 702, the user control system 704, and the auxiliary system 706 can be communicatively coupled via a control line 716, which can represent any wired or wireless communication link, as appropriate for a particular implementation. Therefore, the operating system 702, the user control system 704, and the auxiliary system 706 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0116] Figure 8 An exemplary method 800 for determining the identity of a fluorescent patch present in a scene is shown. Although Figure 8 Exemplary operation according to one embodiment is shown, but other embodiments may omit, add, reorder, combine and / or modify. Figure 8 Any of the steps shown. Figure 8 One or more operations shown can be performed by system 200, any of its components and / or any implementation thereof.

[0117] In operation 802, the fluorescence imaging control system guides an illumination source to illuminate the scene with fluorescence excitation illumination, which is configured to excite fluorophores present at the scene. Operation 802 can be performed in any of the manner described herein.

[0118] In operation 804, the fluorescence imaging control system guides the imaging device to detect fluorescence emitted by the fluorophore in response to excitation of the fluorophore by fluorescence excitation illumination. Operation 804 can be performed in any of the manner described herein.

[0119] In operation 806, the fluorescence imaging control system determines the fluorescence lifetime based on the detected fluorescence. Operation 806 can be performed in any of the manner described herein.

[0120] In operation 808, the fluorescence imaging control system determines the identity of the fluorophore based on the determined lifetime of the fluorescence. Operation 808 can be performed in any of the manner described herein.

[0121] Figure 9 An exemplary method 900 is shown for identifying one or more regions of a distinct tissue type in which fluorophores are present. Although Figure 9 Exemplary operation according to one embodiment is shown, but other embodiments may omit, add, reorder, combine and / or modify. Figure 9 Any of the steps shown. Figure 9 One or more operations shown can be performed by system 200, any of its components and / or any implementation thereof.

[0122] In operation 902, the fluorescence imaging control system guides an illumination source to illuminate the scene with fluorescence excitation illumination, which is configured to excite fluorophores present in the tissue at the scene. Operation 902 can be performed in any of the manner described herein.

[0123] In operation 904, the fluorescence imaging control system guides the imaging device to detect fluorescence emitted by the fluorophore in response to excitation of the fluorophore by fluorescence excitation illumination. Operation 904 can be performed in any of the manner described herein.

[0124] In operation 906, the fluorescence imaging control system generates a fluorescence image comprising multiple pixels based on the detected fluorescence. Generating the fluorescence image includes determining the lifetime of the detected fluorescence at each of the multiple pixels based on the detected fluorescence, and applying pseudo-coloring to the multiple pixels based on the lifetime of the detected fluorescence at each of the multiple pixels. Operation 906 can be performed in any of the manner described herein.

[0125] The systems and methods described herein have been described with reference to fluorescence. However, it should be understood that the systems and methods described herein are not limited to fluorescence, but can be applied to any other type of luminescence, including but not limited to photoluminescence (e.g., phosphorescence), electroluminescence, chemiluminescence, mechanoluminescence, and radioluminescence.

[0126] In some examples, a non-transitory computer-readable medium may be provided for storing computer-readable instructions, based on the principles described herein. When executed by a processor of a computing device, the instructions may direct the processor and / or the computing device to perform one or more operations, including one or more operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0127] As used herein, a non-transitory computer-readable medium can include any non-transitory storage medium that contributes to providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by the processor of the computing device). For example, a non-transitory computer-readable medium can include, but is not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory (“RAM”), and optical discs (e.g., compact discs, digital video discs, Blu-ray discs, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0128] Figure 10 An exemplary computing device 1000 is illustrated, which may be specifically configured to perform one or more processes described herein. Any system, unit, computing device, and / or other component described herein may be implemented by computing device 1000.

[0129] like Figure 10 As shown, computing device 1000 may include a communication interface 1002, a processor 1004, a storage device 1006, and an input / output (“I / O”) module 1008 that are communicatively connected to each other via a communication infrastructure 1010. Although Figure 10 An exemplary computing device 1000 is shown, but Figure 10The components shown are not intended to be limiting. Additional or alternative components may be used in other embodiments. A more detailed description will now follow. Figure 10 The components of the computing device 1000 shown.

[0130] Communication interface 1002 can be configured to communicate with one or more computing devices. Examples of communication interface 1002 include, but are not limited to, wired network interfaces (such as network interface cards), wireless network interfaces (such as wireless network interface cards), modems, audio / video connections, and any other suitable interfaces.

[0131] Processor 1004 generally refers to any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, procedures, and / or operations described herein. Processor 1004 can perform operations by executing computer-executable instructions 1012 (e.g., application programs, software, code, and / or other executable data instances) stored in storage device 1006.

[0132] Storage device 1006 may include one or more data storage media, devices, or configurations, and may take any type, form, and combination of data storage media and / or devices. For example, storage device 1006 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data (including the data described herein) may be stored temporarily and / or permanently in storage device 1006. For example, data representing computer-executable instructions 1012 configured to boot processor 1004 to perform any of the operations described herein may be stored in storage device 1006. In some examples, data may be arranged in one or more databases residing within storage device 1006.

[0133] I / O module 1008 may include one or more I / O modules configured to receive user input and provide user output. I / O module 1008 may include any hardware, firmware, software, or a combination thereof that supports input and output capabilities. For example, I / O module 1008 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0134] I / O module 1008 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O module 1008 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content, as may be available for a particular implementation.

[0135] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and alterations can be made thereto, and additional embodiments can be implemented, without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Accordingly, the description and drawings are to be considered illustrative rather than restrictive.

Claims

1. A system comprising: Memory, which stores instructions; as well as A processor, communicatively coupled to the memory and configured to execute the instructions to: During a surgical procedure performed using a computer-assisted surgical system, a lighting source included in the computer-assisted surgical system is guided to illuminate a scene associated with the surgical procedure using fluorescence excitation illumination, the fluorescence excitation illumination being configured to excite fluorophores present at the scene. The imaging device included in the computer-assisted surgical system is guided to detect fluorescence emitted by the fluorophore in response to excitation of the fluorophore by the fluorescence excitation illumination during the surgical procedure, wherein: The imaging device includes a detector having multiple distinct regions, each distinct region being configured to detect the fluorescence emitted by the fluorophore, and Guiding the imaging device to detect the fluorescence includes guiding the plurality of distinct regions to continuously sample the fluorescence within a time period shorter than the fluorescence lifetime to generate a plurality of fluorescence image signals. Each region included in the plurality of distinct regions generates a distinct fluorescence image signal that is included in the plurality of fluorescence image signals. Based on the multiple fluorescence image signals, the lifetime of the fluorescence is determined. Based on the determined lifetime of the fluorescence, the identity of the fluorophore is determined, and The operation of the computer-assisted surgical system is configured during the surgical procedure and based on the identified identity of the fluorophore.

2. The system of claim 1, wherein the processor is configured to determine the identity of the fluorophore by: Access a lookup table, which includes fluorophore data representing the fluorescence lifetime of fluorescence emitted by each of a plurality of fluorophores, including the fluorophores present at the scene, and A specific fluorophore is selected from the plurality of fluorophores, the specific fluorophore emitting fluorescence having a fluorescence lifetime that best matches the determined lifetime of the fluorescence emitted by the fluorophore present in the scene.

3. The system of claim 1, wherein configuring the computer-assisted surgical system includes setting the configuration of the fluorescence excitation illumination based on the identity of the fluorophore.

4. The system of claim 1, wherein configuring the computer-assisted surgical system includes setting the configuration of the imaging device based on the identity of the fluorophore.

5. The system of claim 4, wherein the configuration of the imaging device specifies at least one of the sampling rate of the imaging device, the exposure time of the imaging device, and the gain of the imaging device.

6. The system of claim 1, wherein configuring the computer-assisted surgical system comprises: A fluorescence image is generated based on the detected fluorescence, which is then presented by a display device communicatively coupled to the processor. The display of the fluorescence image is configured based on the identity of the fluorophore.

7. The system according to claim 1, wherein: The fluorescence excitation illumination is also configured to excite additional fluorophores present in the scene, and The processor is also configured to execute the instructions to: The imaging device is guided to detect, during the surgical procedure, the additional fluorescence emitted by the additional fluorophore in response to excitation of the additional fluorophore by the fluorescence excitation illumination. Based on the detected additional fluorescence, the lifetime of the additional fluorescence is determined, and The identity of the additional fluorophore is determined based on the determined lifetime of the additional fluorescence.

8. The system of claim 1, wherein the fluorescence excitation illumination comprises pulsed fluorescence excitation illumination.

9. A non-transitory computer-readable medium storing computer-readable instructions that, when executed by a processor of a computing device, cause the processor to perform a method, the method comprising: During a surgical procedure performed using a computer-assisted surgical system, a fluorescence imaging control system guides an illumination source included in the computer-assisted surgical system to illuminate a scene associated with the surgical procedure using fluorescence excitation illumination, the fluorescence excitation illumination being configured to excite fluorophores present at the scene. The fluorescence imaging control system guides an imaging device included in the computer-assisted surgical system to detect fluorescence emitted by the fluorophore in response to excitation of the fluorophore by the fluorescence excitation illumination during the surgical procedure, wherein: The imaging device includes a detector having multiple distinct regions, each distinct region being configured to detect the fluorescence emitted by the fluorophore, and Guiding the imaging device to detect the fluorescence includes guiding the plurality of distinct regions to continuously sample the fluorescence within a time period shorter than the fluorescence lifetime to generate a plurality of fluorescence image signals, wherein each region included in the plurality of distinct regions generates a distinct fluorescence image signal included in the plurality of fluorescence image signals; The fluorescence imaging control system determines the lifetime of the fluorescence based on the plurality of fluorescence image signals; The fluorescence imaging control system determines the identity of the fluorophore based on the determined lifetime of the fluorescence, and The operation of the computer-assisted surgical system is configured by the fluorescence imaging control system during the surgical procedure and based on the determined identity of the fluorophores.

10. The non-transitory computer-readable medium of claim 9, wherein determining the identity of the fluorophore comprises: Access a lookup table, which includes fluorophore data representing the fluorescence lifetime of fluorescence emitted by each of a plurality of fluorophores, including the fluorophores present at the scene, and A specific fluorophore is selected from the plurality of fluorophores, the specific fluorophore emitting fluorescence having a fluorescence lifetime that best matches the determined lifetime of the fluorescence emitted by the fluorophore present in the scene.

11. The non-transitory computer-readable medium of claim 9, wherein configuring the computer-assisted surgical system comprises: Based on the identity of the fluorophore, the configuration of the fluorescence excitation illumination is set.

12. The non-transitory computer-readable medium of claim 9, wherein configuring the computer-assisted surgical system comprises: The imaging device is configured based on the identity of the fluorophore.

13. The non-transitory computer-readable medium of claim 12, wherein the configuration of the imaging apparatus specifies at least one of the sampling rate of the imaging apparatus, the exposure time of the imaging apparatus, and the gain of the imaging apparatus.

14. The non-transitory computer-readable medium of claim 9, wherein configuring the computer-assisted surgical system comprises: A fluorescence image is generated based on the detected fluorescence, for presentation by a display device communicatively coupled to the processor, and The display of the fluorescence image is configured based on the identity of the fluorophore.

15. The non-transitory computer-readable medium according to claim 9, wherein: Guiding the sampling of fluorescence in each of the plurality of differentiated regions at different times includes guiding the plurality of differentiated regions to continuously sample the fluorescence within a time period shorter than the lifetime of the fluorescence.

16. The non-transitory computer-readable medium according to claim 9, wherein: The fluorescence excitation illumination is also configured to excite additional fluorophores present in the scene, and The method further includes: The fluorescence imaging control system guides the imaging device to detect, during the surgical procedure, the additional fluorescence emitted by the additional fluorophore in response to excitation of the additional fluorophore by the fluorescence excitation illumination. The fluorescence imaging control system determines the lifetime of the additional fluorescence based on the detected additional fluorescence, and The fluorescence imaging control system determines the identity of the additional fluorophore based on the determined lifetime of the additional fluorescence.

17. The non-transitory computer-readable medium of claim 9, wherein the fluorescence excitation illumination comprises pulsed fluorescence excitation illumination.

Images