Combining molecular chemical imaging and RGB imaging

Abstract

Disclosed herein are methods, systems, and computer program products for fusing molecular chemical imaging (MCI) and red-green-blue (RGB) images. A sample is illuminated with illumination photons that interact with the sample and are used to form an MCI image and an RGB image. The MCI image and the RGB image are fused by a mathematical operation to generate an RGB image with a detection overlay.

Description

Content of disclosure

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 909,525, filed October 2, 2019, the contents of which are incorporated herein by reference in their entirety.

[0002] [Disclosure Field] Molecular chemical imaging (MCI) is a powerful technique for analyzing organic, inorganic, and biological samples of interest, but it does have its drawbacks. One drawback is that real-time or near real-time imaging has not been achieved in many implementations due to limitations in imaging hardware. Another drawback is that the MCI images generated can be difficult to interpret when used alone. This is especially important in biological or medical applications. There is a continuing need to fuse information from multiple images into a single image containing greater information and contrast.

[0003] 〔overview〕 The present disclosure contemplates various embodiments of imaging techniques that fuse two or more images generated from a sample of interest.

[0004] In one embodiment, there is a method of fusing images, the method including: illuminating a sample with illumination photons; acquiring a first sample image from interaction photons interacting with the sample and traveling to a first camera chip; acquiring a second sample image from interaction photons interacting with the sample and traveling to a second camera chip; and fusing the first sample image and the second sample image by weighting the first sample image and the second sample image, wherein the weighting of the first sample image and the second sample image is performed by one or more of Image Weighted Bayesian Fusion (IWBF), Partial Least Squares Discriminant Analysis (PLS-DA), Linear Regression, Logistic Regression, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naive Bayes, Neural Network, or Linear Discriminant Analysis (LDA), thereby generating a fusion score image.

[0005] In another embodiment, the method also includes detecting glare in each of the first sample image and the second sample image, and not classifying portions of the first sample image and the second sample image identified as glare.

[0006] In another embodiment, the method further includes normalizing the intensity of the first sample image and the second sample image.

[0007] In another embodiment, the first sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR, and the second sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR.

[0008] In another embodiment, the first sample image is RGB and the second sample image is Vis-NIR.

[0009] In another embodiment, the first sample image is RGB that is converted to Hue, Saturation, Value (HSV), and the value dimension of the first sample image is omitted.

[0010] In another embodiment, the second sample image is Vis-NIR.

[0011] In one embodiment, there is a system for fusing images, the system including: an illumination source configured to illuminate a sample with illumination photons; a first camera chip configured to acquire a first sample image from interaction photons interacting with the sample; a second camera chip configured to acquire a second sample image from interaction photons interacting with the sample; and a processor that, during operation, fuses the first sample image and the second sample image by weighting the first sample image and the second sample image, wherein the weighting of the first sample image and the second sample image is performed by one or more of Image Weighted Bayesian Fusion (IWBF), Partial Least Squares Discriminant Analysis (PLS-DA), Linear Regression, Logistic Regression, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naive Bayes, Neural Network, or Linear Discriminant Analysis (LDA), thereby generating a fusion score image.

[0012] In another embodiment, the processor detects glare in each of the first sample image and the second sample image and does not classify portions of the first sample image and the second sample image identified as glare.

[0013] In another embodiment, the processor normalizes the intensities of the first sample image and the second sample image.

[0014] In another embodiment, the first sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR, and the second sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR.

[0015] In another embodiment, the first sample image is RGB and the second sample image is Vis-NIR.

[0016] In another embodiment, the first sample image is RGB, the processor converts the first sample image from RGB to hue, saturation, and value (HSV), and the processor omits the value dimension of the first sample image.

[0017] In another embodiment, the second sample image is Vis-NIR.

[0018] In one embodiment, there is a computer program product for fusing images embodied by instructions on a non-transitory computer-readable storage medium, which, when executed by a processor, cause an illumination source to illuminate a sample with illumination photons; a first camera chip to acquire a first sample image from interaction photons that have interacted with the sample; a second camera chip to acquire a second sample image from interaction photons that have interacted with the sample; and the processor to fuse the first sample image and the second sample image by weighting the first sample image and the second sample image, wherein the weighting of the first sample image and the second sample image is performed by one or more of image-weighted Bayesian fusion (IWBF), partial least squares discriminant analysis (PLS-DA), linear regression, logistic regression, support vector machine (SVM), relevance vector machine (RVM), naive Bayes, neural network, or linear discriminant analysis (LDA), thereby generating a fusion score image.

[0019] In another embodiment, the instructions cause the processor to detect glare in each of the first sample image and the second sample image and not classify portions of the first sample image and the second sample image identified as glare.

[0020] In another embodiment, the instructions cause the processor to normalize the intensity of the first sample image and the second sample image.

[0021] In another embodiment, the first sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR, and the second sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR.

[0022] In another embodiment, the first sample image is RGB and the second sample image is Vis-NIR.

[0023] In another embodiment, the first sample image is RGB, the processor converts the first sample image from RGB to hue, saturation, and value (HSV), and the processor omits the value dimension of the first sample image.

[0024] In another embodiment, the second sample image is Vis-NIR.

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the written description, serve to explain the principles, characteristics and features of the invention. [Brief description of the drawings]

[0026] [Figure 1] 1 illustrates one embodiment of real-time detection by registering an RGB image with an MCI image. [Diagram 2] 13 illustrates another embodiment of real-time detection by fusing an RGB image with an MCI image.

[0027] Detailed Description The present disclosure is not limited to the particular systems, methods, and computer program products described, as these may vary, and the terminology used in the description is for the purpose of describing the particular versions or embodiments only, and is not intended to limit the scope.

[0028] As used in this document, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Nothing in this disclosure should be construed as an admission that the embodiments described in this disclosure are not entitled to antedate such disclosure by virtue of prior invention. As used in this document, the term "including" means "including, but not limited to."

[0029] The embodiments described below are not intended to be exhaustive or to limit the teachings to the precise forms disclosed in the following detailed description, but rather the embodiments are chosen and described so that others skilled in the art can appreciate and understand the principles and practices of the present teachings.

[0030] The present disclosure contemplates a system, method, and computer program product designed to illuminate a sample with illumination photons, collect interaction photons from the sample by a camera chip, generate two or more sample images from the interaction photons collected and imaged by the camera chip, and fuse the two or more sample images to generate a target score image. The target score image is generated by applying mathematical operations to the two or more sample images to fuse the two or more sample images. The target score image has greater contrast and information than is possible in any one of the two or more sample images formed from the interaction photons. Further details of the present disclosure are provided below.

[0031] lighting source The illumination source can be any source useful for providing the required illumination while meeting other ancillary requirements such as, without limitation, power consumption, emission spectrum, packaging, heat output, etc. In some embodiments, the illumination source is an incandescent lamp, a halogen lamp, a light emitting diode (LED), a quantum cascade laser, a quantum dot laser, an external cavity laser, a chemical laser, a solid state laser, an organic light emitting diode (OLED), an electroluminescent device, a fluorescent lamp, a gas discharge lamp, a metal halide lamp, a xenon arc lamp, an induction lamp, or any combination of these illumination sources. In some embodiments, the illumination source is a tunable illumination source, which means that the illumination source can be selected to be monochromatic and within any desired wavelength range. The selected wavelength of the tunable illumination source is not limited and may be any passband within the ultraviolet (UV), visible (VIS), near infrared (NIR), visible-near infrared (VIS-NIR), short wave infrared (SWIR), extended short wave infrared (eSWIR), near infrared-extended short wave infrared (NIR-eSWIR), mid wave infrared (MIR) and long wave infrared (LWIR) ranges.

[0032] The above ranges of light correspond to wavelengths of about 180 nm to about 380 nm (UV), about 380 nm to about 720 nm (VIS), about 400 nm to about 1100 nm (VIS-NIR), about 850 nm to about 1800 nm (SWIR), about 1200 nm to about 2450 nm (eSWIR), about 720 nm to about 2500 nm (NIR-eSWIR), about 3000 nm to about 5000 nm (MIR), or about 8000 nm to about 14000 nm (LWIR). The above ranges may be used alone or in any combination of the listed ranges. Such combinations include adjacent (contiguous), overlapping, and non-overlapping ranges. Combining ranges can be achieved by including multiple light sources, by filtering the light sources, or by adding at least one component such as a phosphor and / or a quantum dot that converts high energy emissions, such as UV or blue light, to lower energy light having a longer wavelength.

[0033] sample Illumination photons interact with the sample after being emitted from the illumination source. The sample can be, but is not limited to, any chemical or biological sample where it is desired to know the location of a region of interest relative to the entire sample. In some embodiments, the sample is a biological sample and the illumination photons are used to determine the boundary between a tumor and surrounding non-tumor cells. In some embodiments, the sample is a biological sample and the photons are used to determine the boundary between tissue experiencing blood restriction and tissue experiencing blood perfusion. In some embodiments, the sample is a biological structure and the illumination photons are used to determine the boundary between one biological sample and another biological sample.

[0034] Examples of biological samples include ureters, nerves, blood vessels, lymph nodes, healthy organs, organs experiencing blood restriction, organs experiencing blood perfusion, and tumors. In some embodiments, the biological sample is located within a living organism, i.e., an "in vivo" biological sample. In some embodiments, the sample is not located within a living organism, i.e., an "in vitro" biological sample. In some embodiments, the illumination photons are used to distinguish the biological sample from other structures. In some embodiments, the illumination photons are used to distinguish one biological sample from another biological sample.

[0035] Camera Chip The present disclosure contemplates that there is at least one camera chip that collects and images the interacting photons. In some embodiments, the at least one camera chip is characterized by the wavelengths of light it can image. The wavelengths of light that can be imaged by the camera chip are not limited and include ultraviolet (UV), visible light (VIS), near infrared (NIR), visible-near infrared (VIS-NIR), shortwave infrared (SWIR), extended shortwave infrared (eSWIR), and near infrared-extended shortwave infrared (NIR-eSWIR). These classifications correspond to wavelengths of about 180 nm to about 380 nm (UV), about 380 nm to about 720 nm (VIS), about 400 nm to about 1100 nm (VIS-NIR), about 850 nm to about 1800 nm (SWIR), about 1200 nm to about 2450 nm (eSWIR), and about 720 nm to about 2500 nm (NIR-eSWIR). The above ranges may be used alone or in any combination of the listed ranges. Such combinations include adjacent (contiguous), overlapping, and non-overlapping ranges. Combinations of ranges may be achieved by including multiple camera chips, each sensitive to a particular range, or a single camera chip that can sense multiple different ranges by including a color filter array.

[0036] In some embodiments, at least one camera chip is characterized by the material from which it is made. The material of the camera chip is not limited and can be selected based on the wavelength range that the camera chip is expected to detect. In such an embodiment, the camera chip includes silicon (Si), germanium (Ge), indium gallium arsenide (InGaAs), platinum silicide (PtSi), mercury cadmium telluride (HgCdTe), indium antimony (InSb), colloidal quantum dots (CQDs), or any combination thereof.

[0037] In some embodiments, the camera chip comprises a color filter array for generating an image. The design of the filter array is not limited. When used in the context of a camera chip, the term "filter" should be understood to mean that the light being referenced can pass through the filter. For example, a "green filter" is a filter that appears green to the human eye by filtering only light having a wavelength between about 520 nm and about 560 nm, which corresponds to the visible color green. A similar "NIR filter" passes only near-infrared light (NIR). In some embodiments, the filter is a color filter array positioned over the camera chip. Such color filter arrays are diverse in design, but all are related to the color mosaic filters of the original "Bayer" filter. Color filter arrays include BGGR, RGBG, GRGB, RGGB, RGBE, CYYM, CYGM, RGBW (2x2), RGBW (2x2 diagonal colors), RGBW (2x2 paired colors), RGBW (2x2 vertical W), and X-TRANS (available from Fujifilm Corporation of Tokyo, Japan). X-TRANS sensors have a large 6x6 pixel pattern that includes RGB tiles on every horizontal and vertical line to reduce Moiré effect artifacts. In the table, B corresponds to blue, G to green, R to red, E to emerald, C to cyan, Y to yellow, and M to magenta. W corresponds to the "white" or solid color tile, which is explained further below.

[0038] The W or "white" tile itself includes several configurations. In some embodiments, the W tile does not filter any light, so all light reaches the camera chip. In these embodiments, the camera chip detects all light within a given wavelength range. Depending on the camera chip, this can be UV, VIS, NIR, VIS-NIR, VIS-NIR, VIS-SWIR, or VIS-eSWIR. In some embodiments, the W tile is a filter for VIS, VIS-NIR, NIR, or eSWIR, respectively, allowing only VIS, VIS-NIR, NIR, or eSWIR to reach the camera chip. This can be advantageously combined with any of the camera chip materials or electrical structures listed above. Such a filter array can be useful because it allows for both visible and near infrared light to be detected with one camera chip, and is sometimes referred to as a four-band filter array.

[0039] In further embodiments, the color filter array is omitted and not provided on the camera chip, thereby generating a monochromatic image. In such embodiments, the image generated is based solely on the band gap of the material that composes the camera chip. In other embodiments, the filter is still applied to the camera chip, but only as a monolithic single filter. For example, applying a red filter means that the camera chip generates a monochromatic image that represents the red spectrum. In some embodiments, multiple camera chips are employed, each with a different monolithic single filter camera chip. As an example, a VIS image can be generated by combining three camera chips, each with an R, G, and B filter. In another example, a VIS-NIR image can be generated by combining four camera chips, each with an R, G, B, and NIR filter. In another example, a VIS-eSWIR image can be generated by combining four camera chips, each with an R, G, B, and eSWIR filter.

[0040] In some embodiments, the color array is omitted and the camera chip utilizes vertically stacked photodiodes organized into a pixel grid. Each of the stacked photodiodes responds to light of a desired wavelength. For example, the stacked photodiode camera chip includes R, G, and B layers to form a VIS image. In another embodiment, the stacked photodiode camera chip includes R, G, B, and NIR layers to form a VIS-NIR image. In another embodiment, the stacked photodiode camera chip includes R, G, B, and eSWIR layers to form a VIS-eSWIR image.

[0041] Image Generation Step The present disclosure contemplates that in the first image generation step, the first image is generated by various imaging techniques. In the first image generation step, photons are generated by one or more illumination sources as described above, and the photons travel to a sample. When the photons reach the sample, they interact with the sample. The resulting first interaction photons are thereby emitted from the sample and travel to at least one camera chip. The camera chip thereby generates a first image, which is communicated to a processor.

[0042] Similarly, the present disclosure also contemplates that in a second image generation step, a second image is generated by various imaging techniques. In the second image generation step, photons are generated by one or more illumination sources as described above, and the photons travel to a sample. When the photons reach the sample, they interact with the sample. The resulting second interaction photons are thereby emitted from the sample and travel to at least one camera chip. This causes the at least one camera chip to generate a second image, which is communicated to an image processor.

[0043] The images generated can represent, without limitation, at least one of the following wavelengths: UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, or eSWIR. As used herein, the above ranges of light correspond to wavelengths of about 180 nm to about 380 nm (UV), about 380 nm to about 720 nm (VIS), about 400 nm to about 1100 nm (VIS-NIR), about 850 nm to about 1800 nm (SWIR), about 1200 nm to about 2450 nm (eSWIR), and about 720 nm to about 2500 nm (NIR-eSWIR). In one embodiment, the first image is an RGB image and the second image is a Vis-NIR image.

[0044] Imaging techniques are not limited, and in addition to those described above, imaging may include one or more of laser-induced breakdown spectroscopy (LIBS), stimulated Raman spectroscopy, coherent anti-Stokes Raman spectroscopy (CARS), elastic scattering, photoacoustic imaging, intrinsic fluorescence imaging, labeled fluorescence imaging, and ultrasound imaging.

[0045] Image Fusion Two or more images, including at least a first image and a second image, generated by the interaction of the photons with the sample are fused by an image processor. As mentioned above, the images are not limited and the images generated may be three or more. In one embodiment, the first image is an RGB image and the second image is a Vis-NIR ratiometric image. However, these are not the only possibilities and image fusion can be performed with any two images in the wavelength ranges UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, or eSWIR, whose wavelengths are described throughout this disclosure. Such combinations can be used to generate ratiometric images based on the wavelengths.

[0046] In one embodiment of image fusion, a score image is created first, followed by detection or segmentation. To create the score image, the RGB image and the Vis-NIR image are combined using a mathematical algorithm to create the score image. The score image shows the contrast of the target. For example, in some embodiments, the target appears as a bright "highlight" and the background appears as a dark "shadow". The mathematical algorithm used for image fusion is not limited, and the algorithms include Image Weighted Bayesian Fusion (IWBF), Partial Least Squares Discriminant Analysis (PLS-DA), Linear Regression, Logistic Regression, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naive Bayes, Linear Discriminant Analysis (LDA), and Neural Networks.

[0047] When the mathematical algorithm is IWBF, weighting constants modulate the probability images from each sensor, and the overall target probability is estimated for different combinations of image cross terms. When the IWBF algorithm detects multiple target types, each sensor modality has a single weighting constant for each target type. The selection of each weighting constant for each sensor modality may be accomplished by a variety of techniques. Such techniques include Monte Carlo methods, Receiver Operating Characteristic (ROC) curves, linear regression, neural networks, fuzzy logic, naive Bayes, Dempster-Shafer theory, and combinations of the aforementioned.

[0048] The weighting of each sensor modality for one target type is given by the following formula:

number

[0049] The weighting of each sensor modality for multiple target types is expressed as follows:

number

number

number

number

[0050] In the above formulas 1 to 5, the target type is T, the sensor type is S, the number of sensors is n, and the white image (grayscale consisting of only 1) is W. The detection probability of each target is P T1 , P T2 , P T3 and the weights for combining the images are variables A, B, C, D, E, F, G, H, I, J, K, L.

[0051] The resulting fusion score image or probability image shows contrast enhancement of the target, with higher pixel intensity corresponding to a higher probability that the pixel belongs to the target. Similarly, lower pixel intensity corresponds to a lower probability that the pixel belongs to the target. Detection algorithms using various computer vision and machine learning methods, such as adaptive thresholding and active contours, are applied to the fusion score image to detect targets and find their boundaries.

[0052] In some embodiments, the score image is not generated using the above equation. Instead, a detection or segmentation algorithm is utilized on all N images. Such techniques require a multispectral method in which multiple images are assembled into a hypercube. The hypercube has N images and may include any combination of one or more of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, or eSWIR. In such embodiments, the score image is not generated. Instead, a segmentation algorithm uses all N images, thereby identifying targets. The multispectral method is not particularly limited. In some embodiments, the multispectral method is a spectral clustering method, including one or more of k-means and mean shift methods. In other embodiments, the multispectral detection or segmentation method is a texture-based method that groups pixels together based on similar textures measured across spectral bands using Haralick texture features.

[0053] In some embodiments, the image fusion is generated from images from two cameras. In other embodiments, the image fusion is generated from three cameras. In embodiments where three cameras are used to generate the image fusion, a first camera generates a first tuning state that forms a first molecular chemical image, a second camera generates a second tuning state that forms a second molecular image, and a third camera generates an RGB image.

[0054] In some embodiments where two or more camera chips are included, a stereoscopic image is generated based on images from each of the two or more camera chips. Stereoscopic images are useful because they allow the viewer to perceive depth within the image, increasing the precision and realism of perception. For example, during endoscopic surgery or other similar activities, stereoscopic images are useful for manipulating instruments and performing tasks more safely and accurately than monocular endoscopes, since monocular endoscopes with only one camera chip position cannot provide depth perception. In some embodiments, the stereoscopic image is formed by at least two camera chips, where the camera chips are the same. In some embodiments, the stereoscopic image is formed by at least two camera chips, where the camera chips are different. In any of the above embodiments, the camera chips may have the same color filter array or may have different color filter arrays. In some embodiments, the stereoscopic image is formed by two different camera chips, where only one camera chip is provided with a color filter array and the other camera chip is provided with a monochromatic filter or no filter array at all. Whenever multiple camera chips are provided, the output of each camera chip can be used to generate stereoscopic images by combining or fusing the outputs of each camera chip.

[0055] In a particular embodiment, the illumination in one of the images is not uniform. This often occurs in RGB images and can cause problems when blending the image with other images. To avoid these problems, the images are converted from the RGB color space to the HSV (hue, saturation, value) color space. From the HSV image, only the H (hue) and S (saturation) channels are considered, and V (value) is not included. Therefore, according to the present disclosure, the RGB image can be expressed as an HS image, and alternatively can be called the HS channel.

[0056] As an alternative to the combination of the Vis-NIR image with RGB described above, the Vis-NIR image can be fused with the HS channel instead. This fusion avoids the problem of non-uniform illumination, and the fused image formed from the first sample image and the second sample image shows the contrast of the target. The weighting of the first sample image and the second sample image is performed by one or more of Image Weighted Bayesian Fusion (IWBF), Partial Least Squares Discriminant Analysis (PLS-DA), Linear Regression, Logistic Regression, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naive Bayes, Neural Network, or Linear Discriminant Analysis (LDA).

[0057] In one embodiment, the weighing of the first sample image and the second sample image is performed by PLS-DA for binary or multi-class classification of Vis-NIR images or Vis-NIR images fused with RGB.

[0058] In one useful embodiment illustrated by FIG. 1, a useful method 10 for generating an RGB image 17 with detection overlay is described. First, a sample is imaged by a dual polarization MCI device capable of collecting UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, or eSWIR light in block 11, and the sample is also imaged by a camera chip capable of imaging RGB light in block 12. In block 13, a score image is formed, and the combination of the score image and the RGB image are combined for subsequent image registration. In block 14, detection algorithms are applied to the score image, and the output of these detection algorithms is combined in block 16 with the RGB and MCI registration in overlay detection block 15, thereby generating an RGB image with detection overlay in block 17.

[0059] In another useful embodiment shown by FIG. 2, a useful method 20 for generating an RGB image 28 with detection overlay is described. First, a sample is imaged by a dual-polarized MCI device capable of collecting UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, or eSWIR light in block 21, and the sample is also imaged by a camera chip capable of imaging RGB light in block 22. In block 23, a score image is formed, and the score image is input along with the RGB image so that points on the score image can be aligned with points in the scene in block 24. The RGB image is also fused with the MCI score image in block 25. With the alignment and score image complete, the fused image containing information from the MCI score image and the RGB image is input to a detection algorithm in block 26. Once detection is complete, the fused image with detection is input to block 27 where it is combined with the aligned image from block 24, thereby forming an RGB image with detection overlay in block 28.

[0060] [Example] Example 1 Tests were performed to generate fusion images. To obtain the fusion images, molecular chemistry images were collected and RGB images were collected simultaneously. Collection of molecular chemistry and RGB images was performed within the same in vivo surgical procedure. Molecular chemistry images were collected using an internally developed MCI endoscope, a Hopkins® Telescope 0° NIR / ICG Φ 10 mm available from Karl Storz endoscope collected the RGB endoscope and RGB images were collected.

[0061] Two wavelength images were collected with the MCI endoscope. To fuse the collected MCI and RGB images, the two wavelength images were mathematically combined to generate a ratiometric score image of the target of interest within the in vivo surgical procedure. The MCI and RGB images were then aligned with each other such that each pixel in the MCI image corresponds to the same physical location in the RGB image. This alignment was achieved using a hybrid approach that combines feature-based and intensity-based methods. The feature-based method is first applied to estimate the geometric transformation between the MCI and RGB images. This is achieved by matching KAZE features. KAZE is a multi-scale 2D feature detector and descriptor. The intensity-based method, based on a similarity metric and optimizer, is used to refine the results of the feature-based method. Alignment is achieved by aligning the MCI image to the RGB image using the estimated geometric transformation.

[0062] Next, we perform preprocessing. First, we generate a glare mask by detecting glare in each of the MCI and RGB images. Pixels identified as glare are not classified. Then, the MCI and RGB images are normalized so that the intensities of pixels from these two images are on equal ranges and the intensity does not affect the contribution of each image modality to the fused image.

[0063] After pre-processing, fusion is performed. Using the labeled data generated in the previous training step, the classifier detects pixels that belong to the target of interest. To perform fusion, three frames of the RGB image and the MCI ratiometric score image are input to the classifier. In this example, IWBF is the method used to find the optimal weights of the images that minimize the prediction error in the training set. The weights determined by IWBF in the training set are applied to the images, and the weighted images are then mathematically combined to create the fusion score image. The final fusion score image is then displayed, showing an increase in the contrast of the target compared to the background. This increased contrast can improve the detection performance of the target from the background. In some embodiments, a detection algorithm using computer vision and machine learning methods is applied to the fusion score image to locate or determine the final detection of the target. The final detection is overlaid on the RGB image. The final detection overlaid on the RGB image is particularly useful when the user wants to locate features that are difficult to identify in other situations. In one embodiment, the user is a surgeon who wants to improve the visualization of an organ.

[0064] Example 2 Additional tests were performed to generate fusion images using PLS-DA for binary or multi-class classification. First, RGB and VIS-NIR images were generated from the sample. Specifically, eight images were generated, and the RGB images were aligned to data from a neural composite scene. The RGB images were converted to HSV, which omitted the lightness (luminance) part of the image to avoid uneven illumination of the image. Next, a three-dimensional hypercube was generated. The first dimension of the hypercube was the hue data, the second dimension of the hypercube was the saturation data, and the third dimension of the hypercube was the spectral wavelength. In this example, the spectral wavelengths are 500 nm and 580 nm, which correspond to the nerve recipe.

[0065] We collected hypercubes for scenes containing the following tissue types: intestine, fat, muscle, and nerve, and applied PLS-DA to distinguish the various tissues from each other. Table 1 shows the results when applied to intestine, fat, muscle, and nerve. Events where PLS-DA concluded that the tissue was something other than the tissue in the scene were denoted as misclassifications. For example, in the first column of Table 1, out of a total of six scenes, five were correctly identified as intestinal tissue in the intestinal samples, and one scene was identified as fat tissue in the intestinal samples, resulting in a misclassification rate of 16.7%. As shown in Table 1, even when multiple classes need to be distinguished, the misclassification rate averages 7.1%. [Table 1]

[0066] Similar to the previous experiments, we collected additional hypercubes for scenes containing the following tissue types: intestine, fat, muscle, nerve, and ureter, and applied PLS-DA to distinguish the various tissues from each other. Table 2 shows the results when applied to intestine, fat, muscle, nerve, and ureter. As shown in Table 2, including the ureter did not degrade the two-class performance, maintaining 100% accuracy. [Table 2]

[0067] In the above detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, like numerals typically identify like components unless otherwise indicated by the context. The exemplary embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the various features of the present disclosure, as generally described herein and illustrated in the figures, may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated herein.

[0068] The present disclosure is not limited with respect to the specific embodiments described in this application, which are intended as illustrations of various features. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from the spirit and scope thereof. In addition to those enumerated herein, functionally equivalent methods and apparatuses within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to be included within the scope of the appended claims. The present disclosure is to be limited only by the terms of the appended claims and the full scope of equivalents to which such claims are entitled. It is understood that the present disclosure is not limited to particular methods, reagents, compounds, compositions or biological systems, which can, of course, vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting.

[0069] With respect to the use of virtually any plural and / or singular term herein, those of skill in the art can convert from plural to singular and / or from singular to plural as appropriate to the context and / or application. Various singular / plural permutations may be expressly set forth herein for clarity.

[0070] In general, those skilled in the art will understand that the terms used herein, and particularly in the appended claims (e.g., the body of the appended claims), are generally intended as "open" terms (e.g., the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "includes" should be interpreted as "includes but is not limited to," etc.). Although various compositions, methods, and apparatuses have been described in terms of "including" various components or steps (interpreted to mean "including, but not limited to"), the compositions, methods, and apparatuses can also "consist essentially of" or "consist of" various components and steps, and such terms should be interpreted to define essentially closed member groups. Those skilled in the art will further understand that where a specific number of introduced claim recitations are intended, such intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent exists.

[0071] For example, as an aid to understanding, the following appended claims may include the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed as meaning that the introduction of a claim recitation with the indefinite article "a" or "an" limits any particular claim that includes such an introduced claim recitation to embodiments that include only one such recitation, even if the same claim also includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be construed to mean "at least one" or "one or more"); similarly, the use of definite articles used to introduce claim recitations.

[0072] Furthermore, even when a particular number of introduced claim recitations is explicitly recited, one of ordinary skill in the art will recognize that such recitation should be interpreted to mean at least the recited number (e.g., the mere recitation of "two recitations" without other modifiers means at least two recitations, or more than two recitations). Furthermore, when a convention similar to "at least one of A, B, and C, etc." is used, such a structure is generally intended in the sense that one of ordinary skill in the art would understand the convention (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). When a convention similar to "at least one of A, B, or C, etc." is used, generally such configuration is intended in the sense that one of ordinary skill in the art would understand that convention (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B and C together, etc.). Those of ordinary skill in the art will further appreciate that virtually any disjunctive word and / or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibility of including one of the terms, either of the terms, or both of the terms. For example, the phrase "A or B" will be understood to include the possibilities of "A" or "B" or "A and B."

[0073] Moreover, where features of the disclosure are described in terms of a Markush group, those skilled in the art will recognize that the disclosure is also thereby described in terms of any individual members or subgroups of members of the Markush group.

[0074] As will be appreciated by those skilled in the art, for all purposes, including in terms of providing a written description, all ranges disclosed herein encompass all possible subranges and combinations of subranges thereof. Any recited range can be readily recognized as fully descriptive and allowing the same range to be broken down into at least equal halves, thirds, quarters, fifths, tenths, etc. As a non-limiting example, each range discussed herein can be readily broken down into a lower third, middle third, and upper third, etc. Also, as will be appreciated by those skilled in the art, all language such as "up to," "at least," etc. refers to a range that includes the recited number and can then be broken down into subranges as described above. Finally, as will be appreciated by those skilled in the art, a range includes each individual member. Thus, for example, a group having 1 to 3 cells refers to a group having 1, 2, or 3 cells. Similarly, a group having 1 to 5 cells refers to a group having 1, 2, 3, 4, or 5 cells, etc.

[0075] Various of the above-disclosed and other features and functions, or alternatives thereof, may be combined into many other different systems or applications. Various presently unforeseen or unanticipated alternatives, modifications, variations or improvements may be made thereafter by those skilled in the art, each of which is intended to be encompassed by the disclosed embodiments.

[0076] [Embodiment] (1) A method for fusing images, comprising: Illuminating the sample with illumination photons; acquiring a first sample image from interaction photons that interact with the sample and travel to a first camera chip; acquiring a second sample image from interaction photons that interact with the sample and travel to a second camera chip; fusing the first sample image and the second sample image by weighting the first sample image and the second sample image; Including, The method, wherein the weighting of the first sample image and the second sample image is performed by one or more of Image Weighted Bayesian Fusion (IWBF), Partial Least Squares Discriminant Analysis (PLS-DA), Linear Regression, Logistic Regression, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naive Bayes, Neural Network, or Linear Discriminant Analysis (LDA), thereby generating a fusion score image. (2) detecting glare in each of the first sample image and the second sample image; and 2. The method of claim 1, further comprising not classifying portions of the first sample image and the second sample image identified as glare. (3) The method of embodiment 2, further comprising normalizing the intensities of the first sample image and the second sample image. (4) The method of embodiment 1, wherein the first sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR, and the second sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR. (5) The method of embodiment 4, wherein the first sample image is RGB and the second sample image is Vis-NIR.

[0077] (6) The method of embodiment 1, wherein the first sample image is RGB converted to hue, saturation, value (HSV), and the value dimension of the first sample image is omitted. (7) The method of embodiment 6, wherein the second sample image is Vis-NIR. (8) A system for fusing images, comprising: an illumination source configured to illuminate the sample with illumination photons; a first camera chip configured to acquire a first sample image from interaction photons that have interacted with the sample; a second camera chip configured to acquire a second sample image from interaction photons that have interacted with the sample; and a processor for, during operation, fusing the first sample image and the second sample image by weighting the first sample image and the second sample image; Including, The weighting of the first sample image and the second sample image is performed by one or more of image weighted Bayesian fusion (IWBF), partial least squares discriminant analysis (PLS-DA), linear regression, logistic regression, support vector machine (SVM), relevance vector machine (RVM), naive Bayes, neural network, or linear discriminant analysis (LDA), thereby generating a fusion score image. (9) The system of embodiment 8, wherein the processor detects glare in each of the first sample image and the second sample image and does not classify portions of the first sample image and the second sample image identified as glare. (10) The system of embodiment 8, wherein the processor normalizes intensities of the first sample image and the second sample image.

[0078] (11) The system of embodiment 8, wherein the first sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR, and the second sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR. (12) The system of embodiment 11, wherein the first sample image is RGB and the second sample image is Vis-NIR. (13) The system of embodiment 8, wherein the first sample image is RGB, the processor converts the first sample image from RGB to hue, saturation, and value (HSV), and the processor omits the value dimension of the first sample image. (14) The system of embodiment 13, wherein the second sample image is Vis-NIR. (15) A computer program product for fusing images embodied by instructions on a non-transitory computer-readable storage medium, the instructions, when executed by a processor, comprising: an illumination source illuminating the sample with illumination photons; a first camera chip captures a first sample image from interaction photons that have interacted with the sample; a second camera chip captures a second sample image from interaction photons that have interacted with the sample; the processor blends the first sample image and the second sample image by weighting the first sample image and the second sample image; The computer program product, wherein the weighting of the first sample image and the second sample image is performed by one or more of Image Weighted Bayesian Fusion (IWBF), Partial Least Squares Discriminant Analysis (PLS-DA), Linear Regression, Logistic Regression, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naive Bayes, Neural Network, or Linear Discriminant Analysis (LDA), thereby generating a fusion score image.

[0079] (16) The computer program product of claim 15, wherein the instructions cause the processor to detect glare in each of the first sample image and the second sample image and not classify portions of the first sample image and the second sample image identified as glare. (17) The computer program product of embodiment 15, wherein the instructions cause the processor to normalize intensities of the first sample image and the second sample image. (18) The computer program product of embodiment 15, wherein the first sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR, and the second sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR. (19) The computer program product of embodiment 18, wherein the first sample image is RGB and the second sample image is Vis-NIR. (20) The computer program product of claim 15, wherein the first sample image is RGB, the processor converts the first sample image from RGB to hue, saturation, and value (HSV), and the processor omits a value dimension of the first sample image.

[0080] (21) The computer program product of embodiment 20, wherein the second sample image is Vis-NIR.

Claims

1. 1. A method for fusing images, comprising: Illuminating the sample with illumination photons; acquiring a first sample image from interaction photons that interact with the sample and travel to a first camera chip; acquiring a second sample image from interaction photons that interact with the sample and travel to a second camera chip; fusing the first sample image and the second sample image by weighting the first sample image and the second sample image; Including, the weighting of the first sample image and the second sample image is performed by one or more of Image Weighted Bayesian Fusion (IWBF), Partial Least Squares Discriminant Analysis (PLS-DA), Linear Regression, Logistic Regression, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naive Bayes, Neural Network, or Linear Discriminant Analysis (LDA), thereby generating a fusion score image; A method wherein the first sample image is RGB converted to Hue, Saturation, Value (HSV), and the value dimension of the first sample image is omitted.

2. detecting glare in each of the first sample image and the second sample image; The method of claim 1 , further comprising: not classifying portions of the first sample image and the second sample image identified as glare.

3. The method of claim 2 , further comprising normalizing intensities of the first sample image and the second sample image.

4. The method of claim 1, wherein the second sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR.

5. The method of claim 4, wherein the second sample image is Vis-NIR.

6. 1. A system for fusing images, comprising: an illumination source configured to illuminate the sample with illumination photons; a first camera chip configured to acquire a first sample image from interaction photons that have interacted with the sample; a second camera chip configured to acquire a second sample image from interaction photons that have interacted with the sample; and a processor for, during operation, fusing the first sample image and the second sample image by weighting the first sample image and the second sample image; Including, the weighting of the first sample image and the second sample image is performed by one or more of Image Weighted Bayesian Fusion (IWBF), Partial Least Squares Discriminant Analysis (PLS-DA), Linear Regression, Logistic Regression, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naive Bayes, Neural Network, or Linear Discriminant Analysis (LDA), thereby generating a fusion score image; The system, wherein the first sample image is RGB, and the processor converts the first sample image from RGB to hue, saturation, and value (HSV), and the processor omits the value dimension of the first sample image.

7. 7. The system of claim 6, wherein the processor detects glare in each of the first sample image and the second sample image and does not classify portions of the first sample image and the second sample image identified as glare.

8. The system of claim 6 , wherein the processor normalizes intensities of the first sample image and the second sample image.

9. The system of claim 6, wherein the second sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR.

10. The system described in claim 9, wherein the second sample image is Vis-NIR.

11. 1. A computer program product for fusing images embodied by instructions on a non-transitory computer readable storage medium, the instructions, when executed by a processor, comprising: an illumination source illuminating the sample with illumination photons; a first camera chip acquires a first sample image from interaction photons that have interacted with the sample; a second camera chip acquires a second sample image from interaction photons that have interacted with the sample; the processor blending the first sample image and the second sample image by weighting the first sample image and the second sample image; the weighting of the first sample image and the second sample image is performed by one or more of Image Weighted Bayesian Fusion (IWBF), Partial Least Squares Discriminant Analysis (PLS-DA), Linear Regression, Logistic Regression, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naive Bayes, Neural Network, or Linear Discriminant Analysis (LDA), thereby generating a fusion score image; 11. The computer program product of claim 1, wherein the first sample image is RGB and the processor converts the first sample image from RGB to Hue, Saturation, Value (HSV), and the processor omits a value dimension from the first sample image.

12. 12. The computer program product of claim 11, wherein the instructions cause the processor to detect glare in each of the first sample image and the second sample image and not classify portions of the first sample image and the second sample image identified as glare.

13. The computer program product of claim 11 , wherein the instructions cause the processor to normalize intensities of the first sample image and the second sample image.

14. The computer program product of claim 11, wherein the second sample image is selected from the group consisting of UV, RGB, Vis-NIR, SWIR, Raman, NIR-eSWIR, and eSWIR.

15. The computer program product of claim 14, wherein the second sample image is Vis-NIR.

Images