Camera system and method for identifying nonviable tissue using fluorescent tracer and pulsed high-intensity fluorescent excitation

Abstract

Systems and methods characterize tissue integrity, viability or health of bone and soft tissue through observing perfusion of those tissues, providing guidance to surgeons addressing orthopedic and other trauma. The system exposes a bolus of fluorophore administered to a patient to intense, brief, pulses of infrared excitation light while imaging the fluorophore in tissue with a gated, multi-pulse integrating, CMOS image sensor to obtain a sequence of images. The sequence of images is displayed or processed to map tissue perfusion parameters and / or to map a probability of tissue survival. A method of surgery uses the system during surgery to display to a surgeon the tissue perfusion parameters and / or map of probability of tissue survival, the surgeon then may either revascularize or remove tissue of low probability of survival to improve healing and reduce risk of infection.

Description

CAMERA SYSTEM AND METHOD FOR IDENTIFYING NONVIABLE TISSUE USING FLUORESCENT TRACER AND PULSED HIGH-INTENSITY FLUORESCENT EXCITATIONCROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to U.S. Provisional Patent Application 63 / 456,904 filed April 4, 2023. The entire contents of the aforementioned provisional patent application are incorporated herein by reference.GOVERNMENT RIGHTS

[0002] This invention was made with government support under W81XWH-20-1-0319 awarded by the Department of Defense. The government has certain rights in the invention.BACKGROUND

[0003] Trauma, such as caused by bullet wounds, traffic accidents, and other mishaps, damages tissue, some of which may lose blood perfusion and die or become nonviable; dead and nonviable tissue may become infected. Dead, damaged and non-viable, or badly infected tissue can interfere with wound healing and is often removed during surgical procedures called debridements. When performing debridement, it is desirable to remove as much of the dead, non-viable, or badly infected, tissue as possible to allow healing of the wound, while leaving as much viable tissue as possible because removal of viable tissue can impair healing and result in excessive disfiguring and functional impairment.

[0004] Damaged, nonviable, tissue often has poor perfusion compared to normal, viable, tissue of the same tissue type. In some cases, it also may be possible for surgeons to remove constrictions or repair vessels to revascularize some poorly perfused tissue so it becomes viable and need not be removed.

[0005] An existing system for detecting poorly vascularized tissue, the Stryker SPI-PHI as found at https: / / www.stryker.com / us / en / endoscopy / products / spy-phi.html, images a fluorescent dye, Indocyanine Green (ICG), in tissue over time after a bolus of ICG is administered to a patient. Tissues that receive dye quickly after the bolus is administered are presumably well perfused by blood; while those that receive dye slowly after the bolus is administered are believed less well perfused by blood. Connective tissues like bone andtendons tissues, are, however, normally less well perfused than many other tissues like muscle, liver, and kidney, forcing surgeons to use judgement when determining which poorly perfused tissues are nonviable and which are viable tissues of normally low perfusion tissue types. Observations of tissue perfusion may also indicate tissues that can be made viable by repair of blood vessels that serve those tissues. Ambient light can interfere with detection of ICG fluorescence by the SPY-PHI system, and thus interfere with its use to determine tissue perfusion.

[0006] Some tissues naturally have low perfusion and can be difficult to distinguish from tissue having poor perfusion as a result of trauma, clotting, or infection.

[0007] Many minerals, including some of interest to prospectors and miners, are fluorophores.

[0008] Fluorescence emissions are often of far less brightness than either ambient light or fluorescence stimulus light, making them difficult to see with the unaided eye.

[0009] Fast, time-gated, image sensors on the market today include single-photon avalanche photodiode (SPAD) image sensors, time-gated multi-pulse-integrating (MPI) CMOS image sensors, and time-gated intensified CMOS cameras. One available MPI CMOS image sensor is the Teledyne BORA sensor with 1.3 million pixels.SUMMARY

[0010] Embodiments herein disclose systems and methods for characterizing tissue integrity, viability, or health of bone and soft tissue through observing perfusion of those tissues. Observed perfusion is provided as guidance to practitioners during surgery addressing orthopedic and other trauma. The hardware used forthis purpose is sometimes called "fwdNIR" herein.

[0011] In exemplary methodology, a bolus of fluorophore such as indocyanine green (ICG) is administered to a patient. The system exposes the bolus of fluorophore to intense and brief pulses of infrared excitation light while imaging the fluorophore in tissue with a gated, multi-pulse integrating, CMOS image sensor to obtain a sequence of images. The sequence of images is displayed or processed to map tissue perfusion parameters and / or to map a probability of tissue survival. One method of operation uses the system during surgery to display to a surgeon the tissue perfusion parameters and / or a map of probabilityof tissue survival; the surgeon then may either revascularize or remove tissue of low probability of survival to improve healing and reduce risk of infection.BRIEF DESCRIPTION OF THE FIGURES

[0012] Fig. 1 is a schematic illustration of one camera system 100 examining a wound on a patient, including a block diagram of the camera system having a first lens for imaging with a color image sensor and a second lens for imaging fluorescent emissions with a multiple-pulse-integrating image sensor.

[0013] Fig. 2 is a schematic illustration of another camera system 150 examining a wound on a patient, including a block diagram of the camera system having a single lens and beamsplitter, the beamsplitter diverting infrared light from fluorescent emissions to a multiple-pulse-integrating image sensor and passing white light to a color image sensor.

[0014] Fig. 3 is a schematic illustration of yet another camera system 170 examining a wound on a patient, including a block diagram of a camera embodying an image sensor that includes a deposited filter layer with a Bayer-like tiling pattern of red, green, blue, and fluorescent-emissions-wavelength infrared bandpass filters.

[0015] Fig. 4 is a flowchart of one image capture and image processing method performed using the camera systems of Fig. 1, 2, or 3.

[0016] Fig. 5 is a flow diagram of training a classifier to determine viability of tissue.

[0017] Fig. 6 and Fig. 7 are comparisons of fluorescence readings obtained according to this disclosure in 12-bit intensities and compared to a prior SPI-PHI system in 8-bit intensities with lights on and lights off using a serial dilution of indocyanine green (ICG) + 1% intralipid.

[0018] Fig. 8 is a photograph of fluorescence images obtained with one system made according to this disclosure and following ICG administration in a rat.

[0019] Fig. 9. shows specific absorption of chromophores of interest and the wavebands measured with the PDD arterial input function sensing device.

[0020] Fig. 10A illustrates a curved pulsed LED array for use in the embodiments of Figs.1, 2, or 3; and Fig. 10B illustrates beams obtained with the LED array of Fig. IDA.DETAILED DESCRIPTION

[0021] In embodiments, a nontoxic fluorophore is selected: in particular embodiments the selected fluorophore is indocyanine green (ICG), which has peak absorption at 805 nm (although shorter wavelengths will also excite fluorescence) and peak emissions at about 830 nm. In alternative embodiments other fluorophores may be selected with LED wavelengths and fluorescent emissions wavelength filters appropriate for the selected fluorophore. The selected fluorophore is dissolved in sterile water for injection. When boluses of fluorophore are administered as described below, the dissolved fluorophore is administered intravenously in an instantaneous bolus. When ICG is used, dosage in some embodiments may be 0.1 mg / kg.

[0022] Alternative fluorophores usable with alternative embodiments of the present camera system may include IRDye800 with excitation at 775 nm and emissions at 792 nm; fluorescein sodium, with excitation at 498 nm and emissions at 517 nm; methylene blue with excitation at 668 nm and emissions at 688nm; IRDye680 with excitation at 676 nm and emissions at 693 nm; and IRDye700 with excitation at 689 nm and emissions at 699 nm. An embodiment of the camera system with fluorescence imaging but without perfusion parameter extraction or tissue viability determination may also be useful to image protoporphyrin IX with excitation at 405 nm and emissions at 635 nm following administration of aminolevulinic acid before surgery. Some of these fluorophores emit fluorescence emissions in the infrared and a few emit in visible wavelengths.

[0023] An image processor 102 (Fig. 1, 2, or 3) of the camera system 100, 150 or 170 is coupled through pulsed LED drivers 104 to drive fluorescence excitation wavelength pulsed LEDs 106 selected to emit excitation light primarily of a wavelength suitable to excite fluorescence in the selected fluorophore. In some embodiments, to reduce stray light emitted by the LEDs at fluorescence emissions wavelengths of the fluorophore, a bandpass filter (not shown) is fitted over the pulsed LEDs 106. The excitation light projects onto tissue 134 of the patient 132 and fluorescent emissions light from any fluorophore in the tissue returns to a camera 110, 152 or 172. In alternative embodiments, pulsed LEDs 106 may be pulsed laser diodes. In an alternative embodiment, there are multiple banks of pulsed LEDs, where each bank emits fluorescence excitation light appropriate for at least one fluorophore of potential interest.

[0024] Camera 110 of Fig. 1 has a first lens 112 for imaging with a color or RGB image sensor 114 and a second lens 116 for imaging with a multiple-pulse-integrating (MPI) image sensor 118 synchronized to the pulsed LED driver 104. In the embodiment of Fig. 1, a bandpass filter 120 is provided ahead of second lens 116 to pass fluorescence emissions light while blocking fluorescence excitation wavelength light. Lenses 112, 116 may be simple lenses in some embodiments and compound lenses made from multiple lens elements and / or mirrors in other embodiments. In some embodiments bandpass filter 120 is a fixed- wavelength filter, and in other embodiments intended for compatibility with more than one fluorophore, bandpass filter 120 is selected from a filter-changer having a selection of bandpass filters or a tunable filter configurable to provide bandpass filtering at fluorescence emissions of the fluorophore of interest.

[0025] Camera 152 of Fig. 2 has a single lens 154 and beamsplitter 156, which may be a prism or a dichroic mirror. The beamsplitter 156 diverts fluorescence emissions wavelength light through a bandpass filter 158 to a MPI image sensor 160 and passes white light to a color or RBG image sensor 162.

[0026] Camera 172 of Fig. 3 has a single lens 173 and a single MPI CMOS image sensor 174 having deposited on the image sensor 174 a filter 177 with a Bayer-like tiling pattern of red, green, blue, and fluorescent-emissions-wavelength bandpass filters; this sensor with its tiled filters is capable of performing both RGB or color image sensing and fluorescence emissions imaging.

[0027] Whether camera 110, 152, or 172 is provided, image sensors (e.g., image sensor 162) are coupled to provide captured electronic images to image processor 102. Image processor 102 has a memory 125 containing image processing code 127 configured to capture and process the captured electronic images as described herein and to display resulting images on display 130. All three cameras 110, 152, 172 are small enough to be handheld but are typically mounted to available structures like an operating lamp. The pulsed LEDs 106 are in embodiments located near the lenses and image sensors to make a compact unit requiring no fiber bundle between the LEDs and the cameras. In embodiments, image processor 102 and display 130 are typically mounted remotely from cameras 110, 152 or 172 in a location convenient for a surgeon to view them during surgery.

[0028] LED drivers 104 are configured to drive pulsed LEDs 106 with LED-drive pulses of high peak power having sub-microsecond width and, in an embodiment, pulses of approximately 100 nanoseconds width. These LED-drive pulses repeat at a frequency of 10 kilohertz (kHz) or greater, in some embodiments of 100 kHz or greater, and in an embodiment 300 kHz; in an embodiment this drives the pulsed LEDs with a duty cycle of between 1 in 20 to 1 in 50. This low duty cycle permits driving the pulsed LEDs to provide high peak output power while allowing heat to dissipate from the pulsed LEDs. As shown in Fig. 4, the multipulse-integrating image sensors are gated ON only during the brief pulses when power is provided to the pulsed LEDs, so that 95% to 98% of emissions excited by room illumination light and room illumination light reaching the MPI image sensors is excluded from imaging by these sensors by the MPI image sensor gates. Additional room illumination light is excluded from these MPI image sensors by filters 120, 158, 177.

[0029] As shown in the method 200 of Fig. 4, when in operation, tissue 134 of a patient is exposed during surgery, and pre-bolus images are obtained 202. A bolus of fluorophore is administered and detected 204 by a fluorophore concentration monitoring device, or PDD device 133, that is placed on a finger of the patient. The PDD device 133 provides a measure of an arterial input function for the fluorophore useful in later computations.

[0030] Once pre-bolus background and fluorescence images have been obtained and the bolus of fluorophore administered and the image sensor reset, image processor 102 is configured to perform a first sub-cycle of:

[0031] (a) turning ON 206 the pulsed LEDs and the MPI image sensor gate, waiting a brief time that in an embodiment is 100 nanoseconds and in other embodiments less than one microsecond,

[0032] (b) turning OFF 208 the pulsed LEDs and the MPI image sensor gate,

[0033] (c) waiting or delaying 210 as necessary to maintain a proper pulse frequency that in an embodiment is 3.2 microseconds and in other embodiments less than 100 microseconds, and

[0034] (d) repeating 212 until a predetermined number N pulses of the LEDs have been provided. N is a number chosen to give adequate contrast in fluorescent images and is typically between 10 and 1000.

[0035] The MPI image sensor is then read 214, and its image is stored as a frame in a fluorescence image sequence.

[0036] In some embodiments, in order to reduce background-light interference further than possible with rapid gating of MPI image sensor to image fluorescence emissions wavelength light only during pulses of the bright fluorescence stimulus LEDs, a second subcycle begins of

[0037] (a) turning on 220 the MPI image sensor gate without turning on the pulsed LEDs,

[0038] (b) waiting a brief time that in an embodiment is 100 nanoseconds and in other embodiments less than one microsecond,

[0039] (c) turning OFF 222 the MPI image sensor gate,

[0040] (d) waiting or delaying 224 as necessary to maintain a proper pulse frequency that in an embodiment is 3.2 microseconds and in other embodiments less than 100 microseconds, and

[0041] (e) repeating 226 until a predetermined number M pulses of the MPI gate have been provided.

[0042] The MPI image sensor is then read 228, and its image is stored as a frame in a background image sequence. To further reduce interference from room lighting, images of the background image sequence are subtracted 230 from images of the fluorescence image sequence to form difference images. In some embodiments, to compensate for residual fluorophore from prior boluses given in an earlier stage of surgery, a difference between the pre-bolus fluorescence image and pre-bolus background image is also subtracted from the fluorescence image sequence to give a corrected difference image.

[0043] Both the first and second sub-cycles are repeated for a time of between two and five minutes (inclusive), or until a stop button is pressed, until adequate images of the fluorescence image sequence exist to determine tissue perfusion and viability. During image acquisition, the difference images are displayed to the surgeon.

[0044] In a particular embodiment, an image captured using the RGB image sensors is transformed to a black and white image and the difference image is superimposed in color on the black and white image; in another embodiment the black and white image is falsely colored according to the difference image. These difference images allow a surgeon to visualize perfusion of tissue of the patient in real time, or a prospector to see where fluorescent minerals are in rocks being examined. Additionally, display options include either (1) photons / s or (2) corrected fluorescence, if there is a known phantom in the field of view, which can be calibrated before performing the imaging sequence of Fig. 4 orafterwards as a retro calibration. A third option, units of micromoles / milliliter is a near-real time option, where the area under the curve of the arterial input function (AIF) bolus is determined after 30 s, and then divided by the corrected fluorescence values in the tissue. We call this "area-under-the-curve (AUC)-normalized concentration" and corrects for systemic differences in blood volume, cardiac output, etc.

[0045] Using different modeling approaches, parametric maps are generated. This is based on the convolution theory of tracer kinetics. In the general case, this time-dependent concentration, C(t), measured in the tissue can be described as:C(t) = F R(t) * Ca(t) where F is the blood flow, Ca(t) is the arterial input function and R(t) is the impulse response function— the fraction of ICG remaining in the tissue at time, t, for the idealized case that the input function is defined by a Dirac-delta function with unit mass deposited at time zero. Since C(t) is defined for each pixel (therefore, C(x,y,t)), and Ca(t) is measured with the pulse dye densitometer, then FR(x,y,t) can be calculated, and parameters can be extracted according to the models in Table 1. These are then displayed as parametric maps.Table 1. Impulse residue functions for the three models used in an embodiment of the invention.Tc - capillary transit time, E - extraction fraction, kep, k2 -extravascular to intravascular rate constant, EBF - early blood flow, LBF - late blood flow, F - (total) blood flow, TE - appearance time for early phase vasculature, TL - appearance time for late phase vasculature, ME - minimum transit time across early phase vasculature.The PDD Clip for Arterial Input Function (AIF)

[0046] A pulse dye densitometer (PDD) device 133 may be used to provide an arterial input function measurement that can be of use when computing tissue perfusion. The PDD device 133 may be a finger clip with two or three emitter / detector pairs that measure absorption at 805nm and 900 nm (and 660 nm if a third pair is used), or, if a dye other than ICG is used, other absorption wavelengths associated with the dye in use. These absorption wavelengths are determined from the spectra of chromophores of interest in Fig. 9.

[0047] In one embodiment, signal from the three channels is monitored by an AFE 4900 front-end integrated chip made by Texas Instruments. This chip converts the detected photons into a digitized trace of light fluence by wavelength. Pulse densitometry may be achieved by extracting the pulsatile or AC component of the signal of these channels and attributing the pulsatile change in fluence (difference between systolic and diastolic) as attributable to the change in the partial volume of arterial blood due to expansion and contraction of arterioles. This results in a change in absorption of light proportional to the concentration of "absorbers" like hemoglobin, deoxyhemoglobin and indocyanine green, according to Beer-Lambert law. Since two or three wavelengths are measured, the change of ICG concentration can be separated from changes in (de)oxyhemoglobin. Absolute concentration of ICG can be calculated if the patient's total hemoglobin is known or estimated based on population averages.

[0048] In some embodiments, the first appearance of indocyanine green (ICG) at the PDD device 133 is detected by a time-series classifier approach. The raw signals from the pulse dye densitometer are input toa convolutional neural network (CNN) or a recurrent neural network (RNN), in frames of 5 seconds, and the output is one of three labels: before arrival of ICG, after initial arrival of ICG but before peak ICG, and after peak ICG. This improves the accuracy and reliability of measuring the ratio of ICG concentration to hemoglobin concentration, which is used to calculate cardiac output. In another embodiment, the timeseries classifier uses other machine learning classifiers, such as support vector machines (SVMs), decision trees, k-nearest neighbors (k-NNs), or logistic regression classifiers. In yet another embodiment, the time-series classifier uses a combination of different machine learning algorithms to achieve better performance and robustness.

[0049] In another embodiment, the first appearance of ICG is detected by a time-series classifier approach using a long short-term memory (LSTM) neural network. The LSTM network is a type of recurrent neural network (RNN) that can process sequential data and remember long-term dependencies. Raw signals from the PDD are input to input layers in the LSTM network, in frames of 5 seconds, and the output is one of three labels: before arrival of ICG, after arrival but before peak ICG, and after peak ICG. The LSTM network has several layers of LSTM cells, each containing an input gate, a forget gate, an output gate, and a cell state. The input gate decides which values to update in the cell state, the forget gate decides which values to erase from the cell state, the output gate decides which values to output from the cell state, and the cell state stores the long-term information of the sequence2. In one configuration, the LSTM network has four layers of LSTM cells, each with 128 hidden units, followed by a fully connected layer with 64 units and a softmax layer with three units. The network is previously trained using backpropagation through time (BPTT) and stochastic gradient descent (SGD) with a cross-entropy loss function. In another configuration, the LSTM network can have different numbers of layers, hidden units, activation functions, or optimization methods.Deconvolution-reconvolution for live surgical video correction

[0050] In some embodiments, we provide a method for parallelizing deconvolution / reconvolution for live surgical video correction. The method is used to enable real-time adjustment, including background image subtraction and false-color substitution (if desire) of the dynamic contrast-enhanced fluorescence imaging (DCE-FI) data, which in some embodiments suffers from interpretive errors due to variations in the AIF among different patients and regions of interest.

[0051] In embodiments, this method is based on the theoretical assumption that a measured tissue curve Qx(t) is a convolution of a residue function FR(t) with a patientspecific measured AIF Ca,x(t). The residue function FR(t) represents the fraction of dye remaining in the tissue at time t after injection. The method also assumes that the same tissue, if it were delivered dye according to a standard input function Ca,std(t), would have a time-dye concentration curve (TDC) given by the convolution Qstd(t) = FR(t) * Ca,std(t). The method can then convert the AIFs Ca,x(t) and Ca,std(t) into Toeplitz lower-triangularmatrices CAx and CAstd, respectively. A Toeplitz matrix is a matrix in which each descending diagonal from left to right is constant. The method can then decompose CAx into three matrices Ux, Ex, and Vx using singular-value decomposition (SVD).

[0052] The method then calculates the time-dye concentration curve Qstd(t) as a convolution of FR(t) with Ca,std(t), or equivalently as a matrix multiplication Qstd = FR ■ CAstd. The method approximates the inverse of CAx from the first k singular values of Ex, using the truncated singular value decomposition (tSVD) approach, where k is determined empirically based on the signal to noise ratio (SNR) and sampling rate of the data. The method can then solve the equation Qx • CAstd = Qstd • CAx for Qx using tSVD.

[0053] In one embodiment, the method performs the computation of Qstd for each pixel (x,y) using C++ / CUDA versions of the code, which allows for parallel computing on graphics processing units (GPUs). By parallelizing the computation of Qstd for each pixel, the method achieves faster and more efficient deconvolution / reconvolution for live surgical video correction.Machine Learning

[0054] Fluorescent images and parametric maps are used as features in a classification system. In one embodiment, the input images are processed with kinetic models as well as texture-based models such as gray-level co-occurrence matrix (GLCM)-based approaches. This results in a large number of features, some of which may be correlated. Therefore, a dimension reduction method (e.g., principal component analysis) can be used. Once a set of features (or principal components of features) has been isolated, supervised or unsupervised learning approaches can be used to classify the data.

[0055] In one embodiment, the invention builds a classifier based on supervised training. The classification of a particular pixel is based on its proximity to injury or suspicion of being involved in the mechanism of injury, and subsequent outcome of the patient. Therefore, pixels are classified as either normal, suspicious, or damaged by a clinician developing a training set, and then a composite endpoint is used to assess outcome. A “tissue sparing risk probability (TSP)" is calculated with the following:TSP (%) = a x CE, where a = 0, 0.5, and 1 for normal, suspicious, and damaged

[0056] In another embodiment, pre-operative characteristics of the patient can be considered, such that in addition to a CE, there is an additional factor, POR (pre-operative risk):TSP (%) = a x [P CE + (1- ) POR]Where P is the relative weight given to pre-operative risk and composite endpoints.

[0057] TSP can be converted to a categorical label, such as "low, med, and high" risk with defined cut-offs. Thus, a simple three-class supervised learning model such as k-NN, SVM, decision trees, etc. or any number of deep neural networks, or other deep learning methods might be used to train the classifier to directly predict the TSP from the training set.

[0058] Fig. 5 summarizes the learning process of a supervised learning approach to training the classifier, and use of the trained classifier to evaluate a patient. A time sequence of fluorescence images 502 and arterial input function 504 are used with kinetic modeling to generate AIF-corrected fluorescence images 506 and kinetic maps 508. Spatial and temporal features are then extracted 510 of which principal components 512 are used together with an outcome-based label 514 applied by a surgeon as a training set to train classifier 516. When in use with a previously trained classifier 518, the time sequence of fluorescence images and arterial input function 504 are subjected to similar spatial and temporal feature extraction 520 and principal component extraction before applying the classifier 518 to determine tissue-sparing risk 522. Optionally, the classifier may be updated by soliciting 6 to 12 month outcome information 524 and using this outcome information to enhance the training set and update the classifier.

[0059] In another embodiment, the image processor is configured to analyze the fluorescence images and extract features related to tissue survival, such as perfusion, oxygenation, metabolism, and inflammation. A reinforcement learning algorithm is configured to learn from human feedback (such as expert annotations or clinical outcomes) and optimize a policy that maps the extracted features to a prediction of tissue-sparing risk. In some embodiments, the reinforcement learning method is selected from Q-learning, state-action-reward-state-action (SARSA), actor-critic, and policy gradient methods.

[0060] In both embodiments, the image processor is configured to use the display device to show the prediction map of tissue-sparing risk to the surgeon in real time and guide debridement decision-making during surgery.

[0061] a) In some embodiments, different approaches for reinforcement learning with human feedback (RLHF) are used, including Interactive reinforcement learning (IRL), which allows a human teacher to provide feedback to the agent in the form of rewards or guidance signals during the learning process.

[0062] b) Inverse reinforcement learning (IRL), which infers the reward function of the agent from the demonstrations or preferences of a human expert.

[0063] c) Learning from critique (LfC), which enables the agent to learn from binary feedback (such as "good" or "bad") provided by a human critic after each action.

[0064] d) Deep TAMER, which extends LfC to high-dimensional state and action spaces using deep neural networks and temporal difference learning, may also be used in training.

[0065] e) COACH, which leverages corrective feedback (such as "more" or "less") from a human coach to adjust the agent's policy online may be used in training.

[0066] In some embodiments, the system can distinguish between tissues normally having low perfusion, such as bone or cartilage, from tissues poorly perfused because of traumatic injury, clotting, or infection.RESULTS ACHIEVED:

[0067] Demonstration of limits of detection (LoD) and linear range of ICG fluorescence measurements under three levels of ambient light.

[0068] Figs. 6 and 7 depict the relationship between indocyanine green (ICG) concentration and the pixel values detected by the compact system (12-bit range) and a predicate system, the Stryker SPY PHI (8-bit range) in a phantom. Images were acquired with room lights off in Fig. 6 and on in Fig. 7.Figure-of-Merit SPY PHI Our SystemFOV at 30 cm 26.9 cm x 13.6 cm 15.5 cm x 12.0 cmLimits of Detection of ICG 12 nM 3 nMDynamic Range High Ambient 1:100 1:1,000(i.e., extent of Low Ambient 1:80 1:50 linear range)High Ambient 24% 18%Noise FloorLow Ambient 19% 12%Imaging Head 1 lb. 1.1 lb.WeightTotal, with peripherals 45.3 lbs. 5.9 LBRat tissue imaging

[0069] Tail vein injection of 0.1 mg / kg of indocyanine green in a rat was followed by continuous imaging with our system. An image stack was processed using in house developed software. Pixel values were converted to ICG concentration by means of a phantom (top right corner) of known ICG concentration. Change in [ICG] maps are shown in Fig. 8. This is the principal view of the surgeon during the injection, and while parametric maps are processing.Multi-Fluorophore Embodiments.

[0070] E mbodiments of the camera system useable with more than one fluorophore may also be built. For example, in the embodiment of Fig. 1, emissions-wavelength bandpass filter 120 may be removable or may be part of a filter-changer and set appropriately for the fluorophore in use. Similarly, filter 158 of the embodiment of Fig. 2 may be a filter of a filter changer; and RGB-IR filter deposited on the image sensor in the embodiment of Fig. 3 may have a Bayer-like tiling pattern having more than four bandpass filters; a tiling pattern with 6 filters might allow red-green-blue (RGB) color imaging while supporting three different bandpass filters selected for different fluorophores. On the excitation LED side, there may be two or more groups of LEDs 106 driven separately according to the fluorophore in use;further some fluorophores may be excited with an excitation wavelength shorter than their peak absorption wavelength described above so it may be possible to share excitation LEDs among certain fluorophores.

[0071] Such multi-fluorophore embodiments may prove useful during extended surgeries where many repeated determinations of tissue viability must be made. Similarly, multi- fluorophore embodiments can be useful in locating static fluorescence from fluorescent- tagged antibodies and for other, similar, purposes during surgery.In Conclusion

[0072] Features of this invention therefore may include:

[0073] • ultrafast gated camera with pulsed LED to reject background light, and in embodiments image subtraction to reject additional background light.

[0074] • arterial input function acquisition with pulse dye densitometry

[0075] • tracer kinetic modeling using the adiabatic approximation to the tissue homogeneity (AATH) model, the hybrid plug-compartment (HyPC) model, and / or nonparametric deconvolution models such as constrained or regularized (e.g., truncated SVD, Tikhonov SVD, etc.).

[0076] • extraction of spatial, temporal, and dynamic (kinetic model) features, and their use in machine learning models.

[0077] • Predicts and displays the "tissue sparing risk" - i.e., the risk that sparing that tissue will have on patient outcome (like a false negative rate but connects pixel-level data to patient-level outcome).

[0078] • Live feedback to surgeon regarding pixel-level classification providing actionable information during debridement surgery.

[0079] These features provide the following useful benefits:

[0080] (a) to acquire a series of fluorescence emission images of an injected dye bolus, in some embodiments centered around 820 nm.

[0081] (b) to acquire these images in ambient or room lighting, comparable to normal room lighting or diffuse sunlight through a field hospital tent.

[0082] (c) for the acquisition to be performed using a small form-factor camera / light- source system, that is portable, rugged, and operates using battery power.

[0083] (d) for the light-source to comprise of an array of LEDs pulsed in the microsecond range (in a particular embodiment using pulses of width 100 ns, rate 300 kHz)

[0084] (e) for the camera to be gated to the LEDs, so that the LEDs can be overdriven with a low duty cycle, allowing thermal dissipation

[0085] (f) for the images to be read into a laptop or tablet

[0086] (g) for the fluorescence to be corrected for distance, angle, absorption, and other factors that add to the between-subjects variability

[0087] (h) for the device to use a pulse dye densitometer that quantifies the arterial concentration of dye (also called the arterial input function)

[0088] (i) for kinetic parameters to be extracted from these images using one of a simple plug-flow model, adiabatic approximation to the tissue homogeneity model, the hybrid plug compartment model, or a deconvolution method.

[0089] (j) an analytic approach using multivariate logistic regression, support vector machine, or deep learning to use extracted features and / or parameters from the images to classify tissue into that which should be removed and that which should be spared.

[0090] The pulse-dye densitometer) PDD clip, typically used on a patient's finger - is a dye detection clip with excitation light, fluorescent emissions detector. In an embodiment, the PDD detects light at 3 wavelengths. This distinguishes HbO, HB, ICG so it detects boluses of ICG.

[0091] Bolus given then up to 5 min imaging, impulse function computation to determine tissue viability. Imaging with a plurality of pulses gives parametric maps. Images and parametric maps are provided to the classifier that maps tissue viability per pixels as "tissuesparing risk" The classifier identifies and classifies bone separately because bone typically has low perfusion.

[0092] In order to provide even illumination across a flat field of view, the systems illustrated Figs. 1, 2, or 3 use a curved pulsed LED array as shown in Fig. IDA; and Fig. 10B illustrates beams obtained with the LED array of Fig. 10A. Further, a range finder, such as an ultrasonic range finder or a three-dimensional optical imaging device, is provided to determine distance from the camera 110, 152, or 172 to tissue so that fluorophore concentration in tissue can be quantified based on fluorescence emissions intensity and distance; quantification of the fluorophore concentration helps determine perfusion of the tissue and assists in determining tissue viability.Combinations

[0093] The features described in this document may be combined in various ways.Among combinations anticipated by the inventors are:

[0094] A camera system designated A includes a fluorophore-sensing device configured to detect changes in blood concentration of a fluorescent dye; a plurality of pulsed LEDs that emit light of an excitation wavelength of the fluorescent dye; a pulse driver coupled to drive the plurality of pulsed LEDs; a time-gated camera synchronized to the pulse driver and configured to capture fluorescent emissions wavelength images of tissue; and an image processor coupled to receive the fluorescent emissions wavelength images of the tissue.

[0095] A camera system designated AA including the camera system designated A wherein the plurality of pulsed LEDs is covered by a filter adapted to pass light of the fluorescent excitation wavelength and block light of the fluorescent emissions wavelength.

[0096] A camera system designated AB including the camera system designated A or AA wherein the time-gated camera comprises a multi-pulse-integrating, complementary-metal- oxide-semiconductor (CMOS), image sensor.

[0097] A camera system designated AC including the camera system designated A, AA, or AB wherein the pulse driver is configured to drive the plurality of pulsed LEDs with a sequence of sub-microsecond pulses.

[0098] A camera system designated ACA including the camera system designated AC where the sequence of sub-microsecond pulses comprises pulses at a rate of at least 100 kHz.

[0099] A camera system designated AD including the camera system designated A, AA, AB, AC, or ACA wherein the fluorescent emissions wavelength images of tissue includes a sequence of fluorescent emissions wavelength images extending for at least 2 minutes following administration of a bolus of the fluorescent dye to the tissue.

[0100] A camera system designated AE including the camera system designated A, AA, AB, AC, ACA or AD wherein the image processor is configured to extract tissue perfusion parameters at image pixels from the sequence of fluorescent emissions wavelength images.

[0101] A camera system designated AF including the camera system designated A, AA, AB, AC, ACA, AD, or AE wherein administration of the bolus of the fluorescent dye to thetissue is detected by the device configured to detect changes in blood concentration of a fluorescent dye.

[0102] A camera system designated AG including the camera system designated A, AA, AB, AC, ACA, AD, AE, or AF further comprising an RGB color image sensor.

[0103] A camera system designated AH including the camera system designated A, AA, AB, AC, ACA, AD, AE, or AF where the image processor is further configured to use a trained classifier to distinguish tissue of tissue types having normally low perfusion from poorly perfused tissue of tissue types having normally high perfusion.

[0104] A camera system designated AJ including the camera system designated A, AA, AB, AC, ACA, AD, AE, AF, or AH wherein the image processor is further configured to use a trained classifier to estimate a probability of tissue survival.

[0105] A camera system designated AK including the camera system designated A, AA, AB, AC, ACA, AD, AE, AF, AG, AH, or AJ where the device configured to detect changes in blood concentration of a fluorescent dye is configured as a finger clip

[0106] A camera system designated AL including the camera system designated A, AA, AB, AC, ACA, AD, AE, AF, AG, AH, AJ, or AK where the image processor is configured to use a trained classifier to extract a map of probability of tissue survival from at least tissue perfusion parameters and to display the map of probability of tissue survival to a surgeon.

[0107] A camera system designated AM including the camera system designated A, AA, AB, AC, ACA, AD, AE, AF, AG, AH, AJ, AK, or AL where the image processor is configured by a reinforcement learning algorithm configured to learn from human feedback to optimize a policy that maps extracted features to a prediction of tissue survival probability.

[0108] A camera system designated AN including the camera system designated A, AA, AB, AC, ACA, AD, AE, AF, AG, AH, AJ, AK, AL, or AM wherein the human feedback comprises expert annotations or clinical outcomes, and, wherein the reinforcement learning algorithm is selected from the group consisting of Q-learning, SARSA, actor-critic and policy gradient.

[0109] A pulse dye densitometer designated B for measuring indocyanine green (ICG) concentration in an arterial system, comprising:

[0110] two to four emitter / detector pairs configured to emit and detect light signals at two to four wavelengths;

[0111] a processor configured to receive raw signals from the emitter / detector pairs and calculate a ratio of ICG concentration to hemoglobin concentration; and

[0112] a time-series classifier configured to receive the raw signals from the processor and output one of three labels: before arrival of ICG, after arrival but before peak ICG, after peak ICG.

[0113] A pulse-dye densitometer designated BA including the pulse dye densitometer designated B, wherein the time-series classifier is a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0114] A pulse-dye densitometer designated BB including the pulse dye densitometer designated B, wherein the CNN or RNN receives the raw signals in frames of 5 seconds.

[0115] A pulse-dye densitometer designated BC including the pulse dye densitometer designated B, BB, or BA wherein the CNN or RNN is enhanced with machine learning algorithms, such as support vector machines (SVMs), decision trees, k-nearest neighbors (k- NNs), or logistic regression.

[0116] A pulse-dye densitometer designated BD including the pulse dye densitometer designated B, BC, BB, or BA, wherein the time-series classifier is a long short-term memory (LSTM) neural network.

[0117] A pulse-dye densitometer designated BDA including the pulse dye densitometer designated B, wherein the LSTM network consists of several layers of LSTM cells, each containing an input gate, a forget gate, an output gate, and a cell state.

[0118] A pulse-dye densitometer designated BDB including the pulse dye densitometer designated BDA, wherein the LSTM network has four layers of LSTM cells, each with 128 hidden units, followed by a fully connected layer with 64 units and a softmax layer with three units.

[0119] A camera system designated AO including the system designated AL where the image processor is configured to perform deconvolution and reconvolution for live surgical video correction, including: measuring a tissue curve Qx(t) as a convolution of a residue function FR(t) with a patient-specific measured arterial input function (AIF) Ca,x(t); converting Ca,x(t) and a standard input function Ca,std(t) into Toeplitz lower-triangular matrices CAx and CAstd; decomposing CAx into Ux, lx, and Vx using singular value decomposition (SVD); approximating an inverse of CAx from a first k singular values of 2x, where k is determined empirically based on signal-to-noise ratio (SNR) and sampling rate; calculating a time-dye concentration curve (TDC) Qstd(t) as a convolution of FR(t) withCa,std(t); solving the equation Qx ■ CAstd = Qstd • CAx for Qx using truncated SVD (tSVD); and parallelizing computation of Qstd for each pixel (x,y) using C++ / CUDA versions.

[0120] A method of imaging designated C includes providing pulsed fluorescent excitation light to an object; imaging the object with a camera comprising a time-gated multipulse-integrating CMOS image sensor synchronized to pulses of the pulsed excitation light to capture a first sequence of fluorescent emissions wavelength images; obtaining background images of the object; processing the first sequence of fluorescent emissions wavelength images to determine first images of the object by subtracting the background images from the fluorescent emissions wavelength images; and obtaining reflectance images of the object and displaying the reflectance images with superimposed first images.

[0121] A method of surgery designated D includes: administering a first bolus of a fluorescent dye to a patient; providing pulsed fluorescent excitation light to tissue of the patient; imaging the tissue of the patient with a camera comprising a time-gated multipulseintegrating CMOS image sensor synchronized to pulses of the pulsed excitation light to capture a first sequence of fluorescent emissions wavelength images; and processing the first sequence of fluorescent emissions wavelength images to determine first images of perfusion of the tissue of the patient.

[0122] A method of surgery designated DA including the method designated D wherein pulses of the pulsed fluorescent excitation light are of width less than one microsecond.

[0123] A method of surgery designated DB including the method designated D or DA obtaining reflectance images of the tissue of the patient and displaying the reflectance images with superimposed images of perfusion of the tissue of the patient.

[0124] A method of surgery designated DC including the method designated D, DB, orDA, and including processing the reflectance images of the tissue of the patient and the images of perfusion of the tissue of the patient to determine images of probability of survival of tissue of the patient.

[0125] A method of surgery designated DCA including the method designated DC further comprising removing tissue of low probability of survival from the patient according to the images of probability of survival of tissue of the patient.

[0126] A method of surgery designated DD including the method designated DCA, DC,DB, DA, or D, where capturing the first sequence of fluorescent emissions wavelength images and processing the first sequence of fluorescent emissions wavelength images todetermine first images of perfusion of the tissue of the patient is triggered by a fluorophore sensing device that is attached to the patient and senses a first fluorophore bolus in blood of the patient.

[0127] A method of surgery designated DE including the method designated DD, DCA, DC, DB, DA, or D, further including administering a second bolus of the fluorescent dye to a patient; providing pulsed fluorescent excitation light to tissue of the patient; imaging the tissue of the patient with a time-gated camera synchronized to pulses of the pulsed excitation light to provide a second sequence of fluorescent emissions wavelength images; and processing the second sequence of fluorescent emissions wavelength images to determine second images of perfusion of the tissue of the patient.Conclusion

[0128] Changes may be made in the above methods and systems without departing from the scope hereof. It should thus be noted that the matter contained in the above description or shown in the accompanying drawings should be interpreted as illustrative and not in a limiting sense. The following claims are intended to cover all generic and specific features described herein, as well as all statements of the scope of the present method and system, which, as a matter of language, might be said to fall therebetween.

Claims

Claims:

1. A camera system comprising: a fluorophore-sensing device configured to detect changes in blood concentration of a fluorescent dye; a plurality of pulsed LEDs that emit light of an excitation wavelength of the fluorescent dye; a pulse driver coupled to drive the plurality of pulsed LEDs; a time-gated camera synchronized to the pulse driver and configured to capture fluorescent emissions wavelength images of tissue; and an image processor coupled to receive the fluorescent emissions wavelength images of the tissue.

2. The camera system of claim 1 wherein the plurality of pulsed LEDs is covered by a filter adapted to pass light of the fluorescent excitation wavelength and block light of the fluorescent emissions wavelength.

3. The camera system of claim 1 wherein the time-gated camera comprises a multipulse-integrating, complementary-metal-oxide-semiconductor (CMOS), image sensor.

4. The camera system of claim 1 wherein the pulse driver is configured to drive the plurality of pulsed LEDs with a sequence of sub-microsecond pulses.

5. The camera system of claim 4 where the sequence of sub-microsecond pulses comprises pulses at a rate of at least 100 kHz.

6. The camera system of claim 1 wherein the fluorescent emissions wavelength images of tissue comprises a sequence of fluorescent emissions wavelength images extending for at least 2 minutes following administration of a bolus of the fluorescent dye to the tissue.

7. The camera system of claim 6 wherein the image processor is configured to extract tissue perfusion parameters at image pixels from the sequence of fluorescent emissions wavelength images.

8. The camera system of claim 7 wherein administration of the bolus of the fluorescent dye to the tissue is detected by the device configured to detect changes in blood concentration of a fluorescent dye.

9. The camera system of claim 7 further comprising an RGB color image sensor.

10. The camera system of claim 9 where the image processor is further configured to use a trained classifier to distinguish tissue of tissue types having normally low perfusion from poorly perfused tissue of tissue types having normally high perfusion.

11. The camera system of claim 9 wherein the image processor is further configured to use a trained classifier to estimate a probability of tissue survival.

12. The camera system of claims 1, 2, 3, 4, 5, 6, 7 , 8, 9, or 10 where the device configured to detect changes in blood concentration of a fluorescent dye is configured as a finger clip13. The camera system of claim 12 where the image processor is configured to use a trained classifier to extract a map of probability of tissue survival from at least tissue perfusion parameters and to display the map of probability of tissue survival to a surgeon.

14. The camera system of claim 13 where the image processor is configured by a reinforcement learning algorithm configured to learn from human feedback to optimize a policy that maps extracted features to a prediction of tissue survival probability.

15. The camera system of claim 14, wherein the human feedback comprises expert annotations or clinical outcomes, and, wherein the reinforcement learning algorithm is selected from the group consisting of Q-learning, SARSA, actor-critic and policy gradient.

16. A pulse dye densitometer for measuring indocyanine green (ICG) concentration in an arterial system, comprising: two to four emitter / detector pairs configured to emit and detect light signals at two to four wavelengths; a processor configured to receive raw signals from the emitter / detector pairs and calculate a ratio of ICG concentration to hemoglobin concentration; and a time-series classifier configured to receive the raw signals from the processor and output one of three labels: before arrival of ICG, after arrival but before peak ICG, after peak ICG.

17. The pulse dye densitometer of claim 16, wherein the time-series classifier is a convolutional neural network (CNN) or a recurrent neural network (RNN).

18. The pulse dye densitometer of claim 17, wherein the CNN or RNN receives the raw signals in frames of 5 seconds.

19. The pulse dye densitometer of claim 17, wherein the CNN or RNN is enhanced with machine learning algorithms, such as support vector machines (SVMs), decision trees, k- nearest neighbors (k-NNs), or logistic regression.

20. The pulse dye densitometer of claim 17, wherein the time-series classifier is a long short-term memory (LSTM) neural network.

21. The pulse dye densitometer of claim 20, wherein the LSTM network consists of several layers of LSTM cells, each containing an input gate, a forget gate, an output gate, and a cell state.

22. The pulse dye densitometer of claim 20, wherein the LSTM network has four layers of LSTM cells, each with 128 hidden units, followed by a fully connected layer with 64 units and a softmax layer with three units.

23. The camera system of claim 12 where the image processor is configured to perform deconvolution and reconvolution for live surgical video correction, comprising: measuring a tissue curve Qx(t) as a convolution of a residue function FR(t) with a patient-specific measured arterial input function (AIF) Ca,x(t); converting Ca,x(t) and a standard input function Ca,std(t) into Toeplitz lower- triangular matrices CAx and CAstd; decomposing CAx into Ux, lx, and Vx using singular value decomposition (SVD); approximating an inverse of CAx from a first k singular values of lx, where k is determined empirically based on signal-to-noise ratio (SNR) and sampling rate; calculating a time-dye concentration curve (TDC) Qstd(t) as a convolution of FR(t) with Ca,std(t); solving the equation Qx • CAstd = Qstd ■ CAx for Qx using truncated SVD (tSVD); and parallelizing computation of Qstd for each pixel (x,y) using C++ / CUDA versions.

24. A method of imaging comprising: providing pulsed fluorescent excitation light to an object;imaging the object with a camera comprising a time-gated multipulse-integratingCMOS image sensor synchronized to pulses of the pulsed excitation light to capture a first sequence of fluorescent emissions wavelength images; obtaining background images of the object; processing the first sequence of fluorescent emissions wavelength images to determine first images of the object by subtracting the background images from the fluorescent emissions wavelength images; obtaining reflectance images of the object and displaying the reflectance images with superimposed first images.

25. A method of surgery comprising: administering a first bolus of a fluorescent dye to a patient; providing pulsed fluorescent excitation light to tissue of the patient imaging the tissue of the patient with a camera comprising a time-gated multipulseintegrating CMOS image sensor synchronized to pulses of the pulsed excitation light to capture a first sequence of fluorescent emissions wavelength images; processing the first sequence of fluorescent emissions wavelength images to determine first images of perfusion of the tissue of the patient.

26. The method of claim 25 wherein pulses of the pulsed fluorescent excitation light are of width less than one microsecond.

27. The method of claim 25 further comprising obtaining reflectance images of the tissue of the patient and displaying the reflectance images with superimposed images of perfusion of the tissue of the patient.

28. The method of claim 27 further comprising processing the reflectance images of the tissue of the patient and the images of perfusion of the tissue of the patient to determine images of probability of survival of tissue of the patient.

29. The method of claim 28 further comprising removing tissue of low probability of survival from the patient according to the images of probability of survival of tissue of the patient.

30. The method of claim 25, where capturing the first sequence of fluorescent emissions wavelength images and processing the first sequence of fluorescent emissions wavelength images to determine first images of perfusion of the tissue of the patient is triggered by a fluorophore sensing device that is attached to the patient and senses a first fluorophore bolus in blood of the patient.

31. The method of any one of claims 25 to 29, further comprising administering a second bolus of the fluorescent dye to a patient; providing pulsed fluorescent excitation light to tissue of the patient, imaging the tissue of the patient with a time-gated camera synchronized to pulses of the pulsed excitation light to provide a second sequence of fluorescent emissions wavelength images; processing the second sequence of fluorescent emissions wavelength images to determine second images of perfusion of the tissue of the patient.

Images