Methods, systems, and storage media of determining parameters of a desired target image

Abstract

The invention relates to methods, systems, and storage media for determining parameters of a desired target image. The parameters represent aspects of a scene being imaged, in particular representing physical characteristics of the scene. The characteristic being imaged is blood perfusion or oxygen saturation information on a per-pixel basis. The parameters are obtained by collecting a hyperspectral image of lower temporal and spatial resolution, constructing a virtual hypercube of information with higher spatial resolution using spatial-spectral-aware demosaicking processing, and using the virtual hypercube for estimation of the desired parameters at the higher spatial resolution. Alternatively, a joint demosaicking and parameter estimation operation is performed to obtain the parameters. Various white level and spectral calibration operations can also be performed. The iHSI system implementation allows real-time wide-field HSI and responsive surgical guidance in a highly constrained operating room. Two exemplary implementations using state-of-the-art industrial HSI cameras are investigated, using line-scan and snapshot imaging techniques, respectively.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to image and video processing, and in particular to systems and methods for acquiring and processing hyperspectral images acquired in real time (and in some embodiments, images acquired in a medical context). Background Technology

[0002] Many challenging intraoperative decisions that can have life-altering consequences for patients still rely on the surgeon's subjective visual assessment. This is partly because, even with the most advanced surgical techniques available today, it remains impossible to reliably identify critical structures during surgery. The need for more refined, less qualitative, intraoperative wide-field visualization and characterization of tissues during surgery has been demonstrated across multiple surgical specialties.

[0003] As a first example, in neuro-oncology, surgery is often the primary treatment, with the aim of removing abnormal tissue as safely as possible (Gross Total Resection (GTR)). The pursuit of GTR must be balanced with the risk of postoperative morbidity associated with damage to functionally sensitive areas such as critical nerves and blood vessels. During surgery, techniques such as U.S. Patent No. 9,788,906... The navigation solutions disclosed in B2 can map preoperative information (e.g., MRI or CT) onto the patient's anatomy on the operating table. However, navigation based on preoperative imaging does not account for intraoperative changes. Interventional imaging and sensing (such as surgical microscopy, fluorescence imaging, point-based Raman spectroscopy, ultrasound, and intraoperative MRI) can be used independently by the surgeon or as an adjunct to navigation information to visualize the operated tissue. However, tissue differentiation based on existing intraoperative imaging remains challenging due to the strict surgical constraints in the clinical setting (e.g., intraoperative MRI or CT) or imprecise tumor delineation (e.g., ultrasound or fluorescence imaging). In neuro-oncology surgery, there is increasing use of protoporphyrin IX induced by 5-aminolevulinic acid (5-ALA). Fluorescence-guided surgery using PpIX (PpIX) has been used in other areas, including bladder cancer. However, visualization of malignant tissue boundaries is obscured by the accumulation of tumor markers in healthy tissue; it is non-quantitative, partly due to time-varying fluorescence effects and confounding effects of tissue autofluorescence; it is associated with side effects; and it can only be used for specific tumor types, as reviewed in the Neurosurgical Review, 2019, by Suero Molia et al. The vast array of existing techniques aimed at improving neurosurgical tissue differentiation clearly demonstrates that better intraoperative imaging is seen as an opportunity to improve patient outcomes in these challenging procedures.

[0004] As a second example, necrotizing enterocolitis (NEC) is a devastating neonatal disease that usually requires surgical treatment and has potentially significant side effects. NEC is characterized by ischemic necrosis of the intestinal mucosa, leading to perforation, systemic peritonitis, and, in severe cases, neonatal death. NEC occurs in three out of every thousand live births, with 85% of cases occurring in infants with extremely low birth weight. Despite state-of-the-art care, 30% of infants with short bowel syndrome (NEC) die (1500 g), as reviewed in the Journal of the American College of Surgeons, 2014, by Hull et al. Surgical management of NEC includes primary peritoneal drainage, exploratory confirmation surgery, and / or laparotomy with bowel resection. A major challenge for surgeons performing laparotomy for NEC is deciding how much bowel to remove without compromising the infant's chances of recovery, weighing the long-term risks of leaving poorly perfused intestines in situ. Currently, there is no standard image-guided nursing technique for laparotomy of NEC. Therefore, surgical planning of resection relies on the surgeon's judgment, dexterity, and sensory skills. In case of doubt, coarse tissue resection may be used to assess bleeding. Early diagnosis, better monitoring, and improved surgical management are believed to reduce NEC mortality.

[0005] As discussed in Shapey et al., Journal of Biophotonics, 2019, multispectral and hyperspectral imaging (hereinafter collectively referred to as hyperspectral imaging (HSI)) is an emerging optical imaging technique with the potential to transform the way surgery is performed. However, it remains unclear whether current systems can deliver real-time, high-resolution tissue characterization for surgical guidance. HSI is a safe, non-contact, non-ionizing, and non-invasive optical imaging modality, making it attractive for surgical use. By splitting light into multiple spectral bands far beyond what the naked eye can see, HSI carries fine information about tissue properties beyond conventional color information, which can be used for more objective tissue characterization. In HSI, the data collected within a given time frame spans a three-dimensional space consisting of two spatial dimensions and one spectral dimension. These individual three-dimensional frames are commonly referred to as hyperspectral images or hypercubes. The concept of using HSI for medical applications, as illustrated in U.S. Patent No. 6,937,885 B1, has been known and developed for decades. Classically, HSI relies on acquiring the complete hypercube through spatial and / or spectral scanning. Due to the time required for scanning purposes, these methods cannot provide a live view of the hyperspectral image. Recently, small sensors capable of acquiring HSI data in real time (known as snapshot HSI) have been developed. Such snapshot sensors acquire hyperspectral images at video rates, typically achieving approximately 30 hyperspectral frames per second or more by sacrificing spectral and spatial resolution. Instead of acquiring dense hypercubes (i.e., within a scene...) - Each spatial pixel in the plane has fully sampled spectral information. (Direction), snapshot hyperspectral cameras typically use mosaic patterns, as detailed in Pichette et al., Proc. of SPIE, 2017.

[0006] Here, we define hyperspectral imaging systems as real-time if they are capable of acquiring images at such a video rate, i.e., a time-frequency rate suitable for providing live display of hyperspectral imaging information at tens of frames per second.

[0007] As illustrated in Shapey et al., Journal of Biophotonics, 2019, and further elaborated below in consideration of the prior art, while existing HSI systems can capture important information during surgery, they currently do not provide a means to provide a wide field of view and real-time information with sufficient resolution to support surgical guidance.

[0008] Many different acquisition principles have been described for use in medical applications. One approach relies primarily on the sequential filtering of light on the detector side. As an early example, U.S. Patent No. 5,539,517 A proposes an interferometer-based method in which a predetermined linear combination of spectral intensities is captured sequentially by scanning. Around the same time, U.S. Patent No. 6,937,885 B1 proposes sequentially acquiring HSI data using tunable filters (such as liquid crystal tunable filters (LCTFs)) combined with existing knowledge of the desired tissue response, to acquire data according to a given diagnostic protocol. U.S. Patent No. 8,320,996 B2 improves upon programmable spectral separators such as LCTFs to acquire spectral bands one after another, extracting information relevant to a specific diagnostic protocol, and proposes projecting a summarizing pseudo-color image onto the imaged region of interest. In EP Patent Application No. 2,851,662 A2, a slit aperture coupled with a dispersive element and mechanical scanning is used to acquire spectral imaging information sequentially. Because these methods rely on sequential acquisition, they are not directly suitable for real-time wide-field-of-view imaging. Furthermore, none of these works propose a means to improve the resolution of the captured HSI.

[0009] Besides filtering the light at the detector end, HSI for medical applications has also been explored using filtered excitation light. As a first example, U.S. Patent Application No. 2013 / 0245455 A1 discloses an HSI setup in which multiple LED sources are switched on in a specific sequence to sequentially acquire multiple spectral bands. In a similar approach, WO Patent Application No. 2015 / 135058 A1 discloses an HSI system that requires optical communication between a remote light source and a spectral filtering device to scan through a set of illumination filters. Like their detection-filtering counterparts, these systems are unsuitable for real-time imaging and do not provide solutions for improving HSI resolution.

[0010] Still in the medical field, HSI data sources have been integrated into more complex setups, some intended to provide pathology-related discriminative information. US Patent Application No. 2016 / 0278678 A1 relies on projected spatially modulated light for depth-resolved fluorescence imaging combined with hyperspectral imaging. US Patent No. 10,292,771 B2 discloses a surgical imaging system that potentially includes an HSI device and utilizes a treatment-specific surgical port to reduce port reflection. In US Patent No. 9,788,906 B2, hyperspectral imaging is used as a potential information source to detect stages of a medical procedure and the imaging device is configured accordingly. HSI-derived tissue classification is disclosed in EP Patent Application No. 3545491 A1, where clustering is used to assign the same classification to all pixels belonging to the same cluster. Tissue classification based on HSI data is also proposed in WO Patent Application No. 2018 / 059659 A1. Despite potential interest in their use during surgery, none of these imaging systems have proposed means to acquire real-time HSI or improve the resolution of HSI images, nor have they proposed means to generate high-resolution tissue characterizations for classification maps.

[0011] While HSI has been investigated for assessing a variety of clinical conditions, such as peripheral vascular disease

[11] , retinal ophthalmopathy

[12] , hemorrhagic shock

[13] , foot ulcer healing in diabetic patients

[14] , and cancer detection

[15] , its invivo surgical use has been limited to a few clinical research cases[5]. For example, while the HELICoiD research system

[10] demonstrated promising clinical results for in vivo brain tumor detection

[16] , its size precludes clinical use during surgery. Other systems proposed for intraoperative assessment of tissue perfusion and oxygenation, including those for breast

[17] , oral cancer[7], kidney

[18] , epilepsy

[19] , neurovascular[8], and gastrointestinal surgery[20, 21], further demonstrate the potential of intraoperative HSI (iHSI). However, these are prone to motion artifacts due to the insufficient imaging speed of dynamic scenes during surgery. Recently, two intraoperative systems based on Pushbroom HSI cameras have been proposed, which allow for integration into surgical workflows: in

[22] , the Pushbroom HSI system

[23] is attached to a surgical microscope to capture in vivo neurosurgical data; and in

[24] , a laparoscopic HSI camera is proposed and tested during esophageal surgery. While these systems have shown potential to support surgical workflows, their limited imaging speed is likely to remain a limiting factor for their use during surgery.

[0012] To improve real-time imaging speed, as mentioned above in the section related to image acquisition, recently developed snapshot HSI camera systems have been used to assess brain perfusion in neurosurgery

[25] and to perform preclinical skin perfusion analysis

[26] . However, while snapshot HSI sensors allow for real-time HSI capture using video-rate imaging, spatial resolution is limited and needs to be taken into account in a post-processing step known as demosaicing [27, 28]. Furthermore, previous proposed snapshot iHSI work has not methodologically mapped out and addressed key design considerations to ensure seamless integration into surgical workflows.

[0013] Although various HSI systems have been tested in surgical settings to explore the potential of iHSI, to the best of our knowledge, no HSI system has yet been proposed that allows for stringent clinical requirements, including means of maintaining sterility and ensuring seamless integration into surgical workflows that can provide real-time information for intraoperative surgical guidance.

[0014] Outside the medical field, sensors capable of acquiring HSI data in real time have recently been proposed. In EP patent application No. 3,348,974 A1, a hyperspectral mosaic sensor is proposed, in which spectral filters are interleaved at the pixel level to generate spatially and spectrally sparse but real-time HSI data. Many aberrations are expected in such sensors, and Pichette et al., Proc. of SPIE, 2017, proposed a calibration method to compensate for some of the observed spectral distortions, but did not propose a method to increase spatial resolution. In Dijkstra et al., Machine Vision and Applications, 2019, a learning-based method for acquiring snapshot HSI using a mosaic sensor is proposed. Specifically, a hypercube reconstruction method is proposed. This method focuses only on demosaic / crosstalk correction for hypercube reconstruction. However, they do not disclose parameters for extracting the desired target image from hyperspectral imagery. Although the effects of spectral crosstalk and the sparsity of the sensor are discussed in this work, the combined effects of various distortions are not modeled or directly captured. Simplifying assumptions are used to separate crosstalk correction and upscaling. Alternative methods for capturing snapshot HSI data have been proposed, such as Coded Aperture Snapshot Spectroscopic Imaging (CASSI) proposed in Applied Optics, Wagadarikar et al., 2008. These imaging systems typically include numerous optical components (such as dispersive optics, coded apertures, and multiple lenses), often resulting in impractical shape factors for use in surgery. Similar to mosaic sensors, CASSI systems lead to difficult trade-offs between temporal, spectral, and spatial resolution, and also result in complex and computationally expensive reconstruction techniques. U.S. Patent Application No. 2019 / 0096049 A1 proposes combining learning-based techniques with optimization techniques to reconstruct CASSI-based HSI data. Even with reduced computational complexity, and despite the system's ability to capture raw data in real time, the means for performing real-time reconstruction are not disclosed. Even though sensors such as mosaic sensors and CASSI sensors can be found for use in surgery, it is still necessary to demonstrate how to integrate these sensors into a real-time system capable of displaying high-resolution HSI-derived images and simultaneously providing discriminative images such as tissue characterization or tissue classification for surgical support.

[0015] Existing technologies demonstrate that intraoperative tissue characterization is a problem in many surgical fields, and various methods have been employed to address it. Hyperspectral imaging has shown significant potential in this area. However, to our knowledge, no publicly available method can provide wide-field-of-view and high-resolution tissue-related information derived in real time from hyperspectral imaging during surgery. Therefore, a system and method are needed that allows for improved real-time resolution and correlated tissue characterization with hyperspectral imaging. Summary of the Invention

[0016] Embodiments of the present invention provide a method and system that allow the determination of parameters of a desired target image from hyperspectral imagery of a scene. These parameters can represent various aspects of the scene being imaged, particularly its physical characteristics. For example, in some medical imaging contexts, the characteristic being imaged could be per-pixel blood perfusion or oxygen saturation information. In one embodiment, parameters are obtained by collecting hyperspectral imagery with lower spectral and spatial resolution, then constructing a virtual hypercube with higher spatial resolution information using spatial spectral-aware demosaicing, and then using this virtual hypercube to estimate the desired parameters at the higher spatial resolution. Alternatively, in another embodiment, instead of constructing a virtual hypercube and then performing estimation, a joint demosaicing and parameter estimation operation is performed to obtain parameters directly from the lower spectral and spatial resolution hyperspectral imagery using high spatial resolution. Various white level and spectral calibration operations can also be performed to improve the obtained results.

[0017] In particular, referring to the development of hyperspectral imaging systems, our contributions are fourfold: (i) in contrast to previous work

[29] , we systematically captured a set of design requirements, including functional and technical requirements, that are essential for iHSI systems to provide real-time wide-field HSI information for seamless surgical guidance in highly restricted operating rooms (ORs); (ii) by considering based on Wire scanning and Snapshot Two state-of-the-art industrial HSI camera systems for imaging technologies are presented and evaluated, along with a set of iHSI implementations; (iii) we utilize exemplary iHSI implementations to perform ex vivo animal tissue experiments in a controlled environment to study tissue properties using both camera systems; and (iv) we report the use of a real-time iHSI implementation during an ethically approved clinically feasible case study of hospitalized patients. Figure 5 As part of spinal fusion surgery, this successfully validated our hypothesis that our invention can be seamlessly integrated into the OR without disrupting the surgical workflow.

[0018] In view of the above, from a first aspect, a method is provided for determining parameters of a desired target image from hyperspectral imagery, the method comprising the steps of: capturing a hyperspectral snapshot mosaic image of a scene using a hyperspectral image sensor, the snapshot mosaic image having relatively low spatial resolution and low spectral resolution; de-mosaicing the snapshot mosaic image to generate a virtual hypercube of the snapshot mosaic image data, the virtual hypercube comprising image data having relatively high spatial resolution compared to the snapshot mosaic image; determining a relatively high spatial resolution parameter of the desired target image from the image data in the virtual hypercube; and outputting the determined relatively high resolution parameter as a representation of the desired target image.

[0019] In one example, demosaicing is spatially spectrally aware. For instance, demosaicing may include image resampling (such as linear or cubic resampling) of a snapshot of a mosaic image, followed by the application of a spectral calibration matrix. Furthermore, in another example, demosaicing may include machine learning.

[0020] Alternatively, based on motion compensation between frames, demosaic can be temporally consistent between two or more consecutive frames.

[0021] Another example includes performing a white balance operation on a hyperspectral image sensor before capturing a hyperspectral snapshot mosaic image. In one example, the white balance operation may include: acquiring a reference image separately, the reference image being included during integration time. Dark reference mosaic image and during integration time White reference mosaic image ; and deploying linear models, where, in addition to the objects in integral time The obtained mosaic image It also utilizes the closed shutter to integrate time. and To obtain points for time White reference mosaic image of the reflection patch Dark reference mosaic image and And the white balance operation produces by The given reflection mosaic image.

[0022] In another example, a spatial spectral calibration operation is performed on the hyperspectral image sensor before capturing a hyperspectral snapshot mosaic image. During the calibration operation, the true spectral filter response operator is estimated in a controlled setting. and spatial crosstalk operator To account for parasitic effects during image acquisition.

[0023] Another example could include: acquiring snapshot mosaic image data using collimated light and scanning all images of the target with known, typically spatially constant, spectral feature maps. The characteristics of the hyperspectral image sensor are measured at several wavelengths to obtain the measured system filter response operator. .

[0024] In one example, the steps to determine relatively high spatial parameters also include analyzing pixel-level hyperspectral information to obtain the composition of its unique end-members, characterized by specific spectral feature maps.

[0025] In one example, the step of determining relatively high spatial parameters also includes estimating a tissue characteristic per spatial location (typically a pixel) based on reflectance information from hyperspectral imaging (such as pixel-level tissue absorption information).

[0026] Another example of this disclosure provides a method for determining parameters of a desired target image from hyperspectral imagery, the method comprising the steps of: capturing a hyperspectral snapshot mosaic image of a scene using a hyperspectral image sensor, the snapshot mosaic image having relatively low spatial resolution and low spectral resolution; performing joint demosaicing and parameter estimation from the snapshot mosaic image to determine relatively high spatial resolution parameters of the desired target image; and outputting the determined relatively high resolution parameters as a representation of the desired target image. Within this additional example, all of the white balance and calibration operations described above may also be employed.

[0027] Another aspect of this disclosure provides a system for hyperspectral imaging of a target region, the system comprising: a light source for illuminating the target region; a hyperspectral image sensor configured to capture one or more hyperspectral images of the target region; and an optical scope coupled to the hyperspectral image sensor such that, during use, the hyperspectral image sensor acquires an image of the target generated by the optical scope.

[0028] In one example, the system according to this disclosure is a system for hyperspectral imaging of a target region, the system comprising: a light source for illuminating the target region; and at least one hyperspectral image sensor configured to capture a hyperspectral image of the target region, wherein the system is configured to acquire a plurality of hyperspectral sub-images of the target region on the at least one image sensor.

[0029] In one example, the system according to this disclosure is a system for hyperspectral imaging of a target area, the system comprising: a light source for illuminating the target area; and a hyperspectral image sensor configured to capture a hyperspectral image of the target area, wherein the system is configured to control the switching of the light source at a predetermined frequency.

[0030] In another example, the system according to this disclosure is a system for hyperspectral imaging of a target area, the system comprising: a light source for illuminating the target area; and a hyperspectral image sensor configured to capture a hyperspectral image of the target area, wherein the system includes means for heat dissipation coupled to the hyperspectral image sensor.

[0031] In any of the examples mentioned above, the system may also be configured to determine parameters of a target region from hyperspectral imagery. The system further includes: a processor; and a computer-readable storage medium storing computer-readable instructions that, when executed by the processor, cause the processor to control the system to perform the methods described earlier in this section.

[0032] Another example provides a computer-readable storage medium storing a computer program that, when executed, causes a hyperspectral imaging system according to any of the examples specified above to perform the methods of any of the examples described earlier in this section. Further features and advantages of the invention will become apparent from the appended claims. Attached Figure Description

[0033] Further features and advantages of the invention will become apparent from the following description of embodiments thereof, which are presented by way of example only, and by reference to the accompanying drawings, wherein like reference numerals refer to like parts, and wherein: Figure 1 This is a display illustrating a typical arrangement of filters in a mosaic sensor.

[0034] Figure 2 This is a schematic representation of a mosaic sensor array, which includes the active sensor area of ​​a snapshot mosaic imaging system.

[0035] Figure 3 It shows near-infrared A graph showing the example response of the mosaic sensor.

[0036] Figure 4 This is an example of the display of the molar extinction coefficients of oxyhemoglobin (HbO2) and deoxyhemoglobin (Hb).

[0037] Figure 5 This is a diagram illustrating an example of a sterile imaging system that can be used for real-time hyperspectral imaging.

[0038] Figure 6 This is an exchange diagram representing the steps performed by the computational method during spatial spectral calibration, virtual hypercube reconstruction, and parameter estimation.

[0039] Figure 7 This is a schematic display comparing sparse hyperspectral information obtained from a two-dimensional snapshot mosaic image with information obtained from a three-dimensional hypercube.

[0040] Figure 8 This is a diagram describing the steps of spatial spectral calibration, spatial spectral sensing desmosaic, and parameter estimation via virtual hypercube or acquired snapshot imaging data.

[0041] Figure 9 This is an example illustrating the drawbacks of using methods without spatial spectral awareness to de-mosaic.

[0042] Figure 10 This is an example of a display of different tissue characteristic parameter diagrams extracted from a virtual hypercube.

[0043] Figure 11 This is a diagram of a computer system according to an embodiment of the present invention.

[0044] Figure 12 This is a schematic diagram illustrating implementation methods of line scan and snapscan imaging.

[0045] Figure 13 This is a schematic diagram of a xenon light source with and without a UV filter.

[0046] Figure 14 This is the display of the chessboard experiment settings.

[0047] Figure 15 This is a graph showing an example reconstruction of the chessboard experiment.

[0048] Figure 16 This is a display of the settings for in vitro experiments.

[0049] Figure 17 This is a display of example sequences acquired by a camera that captures HSI data of the spinal cord and small roots during ex vivo imaging.

[0050] Figure 18 This is a comparison of the estimated reflection curves.

[0051] Figure 19 This is a display of the intraoperative HSI setup (iHSI) during a spinal fusion study, with an example being an in vivo snapshot mosaic image. Detailed Implementation

[0052] According to one aspect, the embodiments described herein relate to a computer-implemented method and system for obtaining a hyperspectral image from low-resolution mosaic image data acquired in real time, in order to determine image characteristic information representing some physical properties of the sample being imaged. The method may include the following steps: Acquiring hyperspectral imaging data using medical devices suitable for use in sterile environments The application of data-driven computational models enables the delivery of hyperspectral information at a resolution that is more stringent than any spectral band of the original data.

[0053] Imaging systems used for data acquisition in conjunction with this method and system may include one or more hyperspectral imaging (HSI) cameras and light stimulation provided by one or more light sources. Their application can be combined with instruments such as exoscopes or endoscopes as part of the optical path of the imaging system.

[0054] The imaging system can be handheld, for example, fixed to the operating table by means of a robotic arm or combined with a robotic actuation mechanism.

[0055] The filter can be placed anywhere in the optical path between the origin of the traveling light and its receiver (such as a camera sensor).

[0056] Hyperspectral images can be acquired using imaging devices that capture sparse hyperspectral information, such as by assigning spectral information to each spatial location a number of spectral bands that are strictly fewer than the total number of spectral bands that the imaging system can measure. An example of such a prior art imaging system is shown in US 9857222 B, which describes the use of a filter mosaic for passing through different spectral bands, and a sensor array of pixels configured to detect images at different spectral bands passed by the filters, wherein, for each pixel, the sensor array has a cluster of sensor components for detecting different spectral bands, and the mosaic has corresponding filter clusters for different spectral bands integrated on the sensor components, so that images can be detected simultaneously at different spectral bands.

[0057] Example imaging systems may include, for example Figure 1 The individual shown A mosaic sensor array. An effective sensor area is obtained by creating such an array of individual mosaic sensors (see...). Figure 2 ).each The mosaic sensor integrates 25 filters sensitive to different spectral bands. This results in sparse sampling of hyperspectral information across the effective sensor region, where information for each spectral band is only obtained in each... The region is acquired once and spatially shifted relative to other spectral bands. Images acquired by this sensor array arrangement are referred to as "mosaic" or "snapshot mosaic" images.

[0058] Due to limitations and physical design constraints inherent in hyperspectral camera sensors in practice, parasitic effects can lead to multimodal sensitivity response profiles in filters. Examples of such effects affecting imaging include interference from higher-order spectral harmonics, out-of-band leakage, and crosstalk between neighboring pixels on the camera sensor. Figure 3 Near-infrared (NIR) is shown. Example response curves of a mosaic sensor. It will be clear that additional filters can be used to suppress or emphasize the spectral components of these responses. Natural hypercube reconstruction of snapshot mosaic images obtained by stacking images of band-correlated pixels results in a hypercube representation that is spatially and spectrally distorted, exhibiting low resolution in both the spatial and spectral dimensions. Therefore, snapshot hyperspectral imaging is characterized by high temporal resolution of hyperspectral images contaminated with multi-peak spectral bands, and low resolution in both the spatial and spectral dimensions.

[0059] Here, we disclose a method suitable for obtaining hyperspectral imaging information from snapshot imaging with high temporal and spatial resolution, which provides tissue-related information with a wide field of view and high resolution in real time during surgery.

[0060] White balance and spatial spectral calibration of the acquired hyperspectral snapshot mosaic image can be performed as a preprocessing step using data acquired at the factory or by the user. In some examples, this can be achieved using a single image of a static object (such as a reflector) acquired inside or outside the operating room, or a sequence of images of static or moving objects. In other examples, this can be done by processing specular reflections observed in the acquired image data. Further examples of image calibration may involve image processing due to intentional alterations to imaging device settings (such as the effects of changing filter tuning).

[0061] Reconstructing high-resolution hyperspectral image data from raw low-resolution snapshot mosaic using image processing methods is referred to as “demosaicing” or “upsampling.” Demosaicing can be performed by spatially spectrally upsampling the acquired snapshot mosaic data to obtain a hypercube with a fixed, potentially arbitrary number of spectral band information for all acquired image pixel locations. In some implementations, such demosaicing can also be performed to achieve spatially spectrally upsampling of other high-resolution grids besides the original image pixel locations. In addition to conventional demosaicing methods that achieve spatial upsampling due to, for example, resampling, we propose spatially spectrally-aware upsampling / demosaicing methods that address both spatial crosstalk and spectral parasitic effects, which are particularly important in snapshot imaging systems. This reconstruction will be referred to as a “virtual hypercube.” Simple examples of demosaicing can include image resampling performed independently on each spectral band. In other examples, this can include methods based on inverse problem formulations. Other examples can also include the use of data-driven, supervised, semi-supervised, or unsupervised / self-supervised machine learning methods. These examples can also include computational methods for reconstruction designed for irregular grids. Increased quality or robustness during demosaicing can be achieved by processing video streams of image data. Similar methods can be used to increase the temporal resolution of data visualizations.

[0062] Computational models can be used for parameter estimation based on a virtual hypercube representation. Examples can include estimating tissue properties per spatial location (typically a pixel) based on reflectance information from hyperspectral imaging (such as pixel-level tissue absorption information). More generally, the acquired pixel-level hyperspectral information can be analyzed to obtain its composition of unique endmembers characterized by specific spectral feature maps. A spectral nonmixing algorithm is proposed that estimates the relative abundance of mixed endmembers in pixel spectra to derive surgically guided tissue properties. Relevant examples of endmembers include oxyhemoglobin and deoxyhemoglobin (…). Figure 4 Examples of tissue characteristics exported by relevant end users include per-pixel blood perfusion and oxygenation saturation information.

[0063] Other examples of non-mixing can include the estimation of fluorescence and autofluorescence, which can also be used for quantitative fluorescence.

[0064] On the other hand, virtual hypercube representations can be used to estimate pseudo-red-green-blue (RGB) images or any other dimensionality-reduced images to visualize hyperspectral imaging data.

[0065] On the other hand, virtual hypercubes can also be used to classify pixels based on tissue type, including benign and malignant types. Virtual hypercubes can also be used for semantic segmentation beyond tissue type. This can include the classification of any pixels associated with non-human tissue, such as surgical tools. The resulting segmentation can be used to improve the robustness of tissue parameter estimation or to correct potential image artifacts such as specular reflections.

[0066] In all the examples described here, virtual hypercube estimation and parameter extraction can be performed as two separate steps or they can be performed jointly. The computational model can use algorithms that allow for joint demosaicing and parameter estimation. This approach can be based on inverse problem formulation or on supervised or unsupervised machine learning methods.

[0067] All computer-aided parameter estimates can be associated with uncertainty estimates.

[0068] This disclosure relates to an image processing system and method that allows for online video processing of high-resolution hypercube data from a video stream of sparse, low-resolution mosaic data acquired in real time by a medical device suitable for use in a sterile environment.

[0069] System Description

[0070] Figure 5 An overview of an example sterile imaging system that can be used for real-time hyperspectral imaging is presented. A real-time hyperspectral imaging camera, such as a snapshot hyperspectral camera, is mounted on a sterile optical observation instrument, such as a sterile exoscope, via a suitable adapter. This adapter may also allow zooming and focusing of the optical system and may include additional features such as a mechanical shutter, beam splitter, or filter adapter. In some embodiments, using multiple camera sensors in conjunction with a beam-splitting mechanism can advantageously cover the wavelength range of interest. For ease of presentation, this configuration may continue to be referred to as a hyperspectral imaging camera. The sterile optical observation instrument is connected to a light source (such as a broadband xenon or LED light source) that can provide light at spectral wavelengths suitable for the hyperspectral imaging camera, or excite the fluorophore of interest via a light guide that may be sterile or draped with a sterile drapery. It should be understood that in some embodiments, the light source may also be mounted together with the camera, potentially eliminating the need for a light guide. Filters can be placed anywhere in the optical path between the origin of the traveling light and its receiving end (such as a camera sensor). Such filters can be inserted into a light source using various devices such as filter wheels, which can hold multiple filters or can be embedded in an adapter or endoscope. In some embodiments, filters can be used to eliminate unwanted out-of-band responses, such as the visible light portion of an NIR sensor. Figure 3 The hyperspectral imaging camera is connected to a computing workstation via a data link, such as a cable or wireless communication. Advantageously, power can be supplied to the camera sensor and other power-enabled components (e.g., tunable lenses or filters) mounted on the camera via the same cable as the data link (e.g., connected with Power over Ethernet (PoE)). The workstation processes the acquired hyperspectral imaging information and can display the derived information to the user via a display monitor. In some embodiments, the workstation can be embedded in the camera sensor unit or the display monitor. The visualization information may include the acquired hyperspectral imaging data, or information derived from the hyperspectral imaging data via the computational methods described below, such as RGB images or tissue characteristic information. Different types of information can be overlaid to provide more background to the user. In one example, tissue characteristic information in areas estimated with high confidence can be overlaid on a pseudo-RGB rendering of the captured scene. The sterility of the imaging system can be ensured by covering system components with sterile curtains or by a combination of sterilization and procedural steps to ensure that the connection between sterile and non-sterile components does not compromise the sterility of the operator and the field. One advantageous implementation could be the use of a sterile curtain sealed over a sterile optical observer connected to a sterile light guide for the camera and data cables. The sterile imaging system can be handheld by the user or fixed to an operating table that allows for control of the imaging system's movement or fixation as required by the user during surgery. Controlled movement and fixation of the sterile imaging system can be achieved using a sterile or sterile-curtained robotic arm or robot actuation mechanism. In other implementations, the hyperspectral imaging system can be embedded within a surgical microscope.

[0071] More generally, we will now present the key design requirements for intraoperatively guided HSI suitable for open surgery. These standards will be described and explained in more detail. Figure 5 The iHSI invention is specifically implemented and illustrated in the text.

[0072] Intraoperative HSI system design requirements

[0073] The first design assumption is to facilitate the intraoperative application of HSI camera systems by developing independent, lightweight devices that are independent of or complement surgical microscopes typically used in neurosurgery. In particular, a modular and flexible system design is achieved by ensuring compatibility with surgical telescopes (such as exoscopes

[30] or endoscopes) and surgical microscopes, and suitability for open or endoscopic surgery across surgical specialties. Following this assumption, Tables 1 and 2 provide an overview of the design requirements, including minimum and target requirements, for the implementation of a hyperspectral imaging system for intraoperative surgical guidance. These are divided into (i) functional requirements, i.e., requirements derived from the clinical environment of the operating room (OR) during surgery (Table 1), and (ii) technical requirements, i.e., specifications for the HSI system to deliver high-fidelity imaging data to meet the functional requirements for the listed real-time surgical guidance purposes (Table 2). When objective requirements are not readily available, best estimates are given based on our experience, as outlined below.

[0074] As part of the surgical requirements, ensuring the aseptic nature of the iHSI system is beneficial so that it can be safely manipulated by the surgical team (F1); compliance with standard technical safety specifications is beneficial (F2); lighting and illumination requirements should advantageously not interfere with the surgical workflow (F3); and the device should advantageously be easy to maintain and clean, conforming to standard surgical practice (F4). It should advantageously be securely mounted during the surgical procedure, but handheld devices should advantageously be easy to manipulate, allowing for control of the imaging system's movement and fixation by a single operator without the need for an assistant (F5). The spatial resolution and spectral information captured within the surgical images should advantageously be compatible with the surgical actions (F6), i.e., providing information on a wide field of view covering a minimal area that provides sufficient background for surgical decisions. Additionally, it should advantageously facilitate the ability to perform broader tissue monitoring related to the surgery. The device should advantageously be able to provide critical functional or semantic organizational information and should be able to provide detailed information on multiple characteristics used for comprehensive patient monitoring to improve surgical accuracy and patient safety during the surgical procedure (F7). In the case of neuro-oncology surgery, this may be for defining tissue boundaries to clearly demonstrate the tumor tissue and its relationship to key brain structures such as nerves, blood vessels, or normal brain tissue. Furthermore, the image resolution should be advantageously detailed enough to facilitate spatial differentiation between tissue types within the surgical field of view (F8). Imaging should be advantageously displayed at video rates to facilitate immediate surgeon feedback and integration with seamless workflows at higher video rates, thus allowing for a smoother experience (F9). Accurate visualization of the extracted information is beneficial for surgical guidance, thereby enabling a better user experience using an intuitive display system (F10).

[0075] To ensure surgical safety and asepticity, system maintenance should be advantageously straightforward and should allow for efficient cleaning of system components (T1) using standard antimicrobial surface wipes. Minimum advantageous requirements for the size and weight of the HSI camera are based on estimates obtained through a prototype testing design thinking approach

[31] in

[29] , namely, less than 10 × 10 × 12 cm. 3 (T2) and cameras weighing less than 1.0 kg (T3). For systems smaller than 6×6×8 cm3, a standard sterile curtain can be used to cover the camera to ensure sterility. In addition, all camera edges should be advantageously smoothed to prevent tearing of the sterile curtain and injury to staff (T4). A maximum camera temperature of 40°C advantageously ensures the technical safety of device operation and also reduces dark current to maintain an appropriate signal-to-noise ratio (SNR) during image acquisition (T5). The number of cables used to power the camera and connect to the camera for data should be advantageously kept to a minimum (T6). In order to enable adequate iHSI, a suitable light source should advantageously be available to provide sufficient energy across the effective spectral range of the HSI camera (T7), but should advantageously comply with technical safety and light safety considerations so as not to cause harm to the patient due to light exposure (T8). This includes advantageously complying with the maximum permissible exposure (MPE), especially for ionizing ultraviolet (UV) wavelengths below 400 nm

[32] . The light source setting adjustment should advantageously allow for optimal illumination conditions for acquiring HSI information during the procedure (F3 and T10). In addition to the optimal light intensity setting depending on the surgical scene, this can advantageously include adjusting the filter according to the imaging requirements of the HSI camera to obtain high-fidelity HSI signal measurements. Advantageously, these settings can be adjusted automatically by taking into account dynamic changes in the OR (such as illumination). The static mounting system is the minimum advantageous requirement that allows sufficient intraoperative device manipulation (T9). The camera settings can be advantageously adjusted according to the surgical background to obtain high-fidelity HSI information (T10), including by implementing an adjustable system mount (T9) that enables automatic adjustment of the camera settings (T10). Further improvements in device manipulation can be achieved by meeting advantageous target requirements for camera size and weight (T2, T3).

[0076] High-fidelity tissue information can benefit from the corresponding target tissue being in the imaging field of view and kept in focus during HSI acquisition. This may require refocusing during surgery, which can be achieved using manual or automatic focusing setups (T11). A fixed working distance (WD) between 200 mm and 300 mm (T12), a fixed field of view (FOV) between 40 mm and 60 mm (T13), and a depth of field (DOF) of at least 20 mm (T14) are the minimum favorable requirements for iHSI

[33] , but more favorable scenarios include systems that can change WD, FOV, and DOF to maximize compatibility with current surgical visualization systems

[34] . The number of spectral bands, spectral range, and spatial image sharpness depend heavily on the clinical application providing key functionalities and / or semantic features. Based on a review of previous literature and our own experience, providing several dozen well-defined spectral bands is beneficial for achieving significant improvements over standard RGB imaging. Based on the availability of state-of-the-art snapshot HSI sensors in the industry (refer to Table 3), exemplary embodiments that achieve the advantageous requirements of an iHSI system during surgery can provide 16 spectral bands (T15) and a spectral range of at least 160 nm (T16). Other exemplary embodiments, while at a lower frame rate, still utilize at least 100 spectral bands with a spectral coverage of at least 500 nm (refer to Table 3) and can advantageously achieve superior tissue differentiation capabilities. In the case of providing information with an advantageous accuracy of at least 1 mm, exemplary embodiments will achieve at least 3 pixels per millimeter. Exemplary embodiments that achieve the minimum advantageous requirements and the target FOV advantageous requirements include imaging grids of at least 120 × 120 and 450 × 450. However, substantially higher resolutions are possible based on currently available HSI sensor technology. Therefore, exemplary embodiments advantageously target high-resolution (1920 × 1080 pixels) and ultra-high-resolution (3840 × 2160) resolutions respectively for the minimum advantageous requirements and the target advantageous requirements (T17). Image calibration advantageously supports the generation of interpretable HSI data. Typical implementations involve acquiring white and dark reference images for white balance, taking into account ambient light and specific camera settings (T18). As an example, this can typically be achieved by acquiring images using white reflectance patches and with the shutter closed, respectively. However, for surgical guidance in OR, calibration data should be advantageously available without disrupting the clinical workflow.

[0077] Minimum favorable imaging rates benefit from being fast enough to provide real-time information suitable for surgical decision-making without interfering with the surgical workflow (T19). Based on the processing speed of the human visual system, image visualization rates faster than 7 frames per second (FPS) are advantageous

[35] . In some scenarios with static scenes, an image acquisition rate of a few seconds per image per surgical scene can be sufficient to provide critical information to the surgical team. However, iHSI suitable for real-time image-guided surgery should be advantageously able to provide video-rate imaging to provide a live display of tissue information, which also allows for dynamic scenes during surgery. Intraoperative HSI System Design

[0078] By following the above system design requirements, we propose... Figure 5 The iHSI system is specifically implemented and illustrated in the diagram. The HSI camera can be advantageously connected to a sterile optical observer via a suitable eyepiece adapter. The sterile optical observer can be advantageously connected to a light source via a sterile light guide.

[0079] The following are some specific aspects, features, and advantages of embodiments according to this disclosure.

[0080] 1. Combining a surgical exoscope with an HSI camera for real-time tissue characterization enhances the surgeon's vision and aids in intraoperative decision-making. In some implementations, the sterile optical observer can be an exoscope, but it can also be an endoscope or surgical microscope.

[0081] 2. A filter is embedded in the HSI camera to filter the effective signal closest to the HSI camera sensor, thereby enabling high-fidelity data capture. In some implementations, the filter can be advantageously added to the eyepiece adapter to obtain high-fidelity hyperspectral images while allowing for a compact optical system.

[0082] 3. Use a custom filter arrangement to address quantum efficiency differences among individual spectral band sensors and obtain a balanced sensor response across all spectral band sensors. In some examples, this may include using a custom filter arrangement to address quantum efficiency differences among individual spectral band sensors and obtain a balanced sensor response across all spectral band sensors for signal processing.

[0083] 4. Filter embedding allows for in-process image calibration during surgery. Given a target with known reflectivity, the optical path can be intentionally altered by adjusting the filter arrangement used for in-process image calibration during surgery. In some implementations, these filter adjustments can be performed using a filter wheel within the light source.

[0084] 5. Use an infinity correction module to maintain the same beam focus when multiple filters are arranged at the HSI camera sensor plane. In some examples, the infinity correction module can be used to generate a parallel beam between the objective lens and the barrel lens. This will allow the addition of parallel planar optical components (e.g., filters) without changing the parfocal point, and therefore without causing any image shift.

[0085] 6. Custom-designed accessories for telescopes to interlock optical system components to prevent slippage, rotation, or any other changes, thereby preventing undesirable changes in the optical characteristics of the optical system during surgery. Optical system components, including filters, telescopes, and optical lenses, can be mechanically locked to prevent undesirable changes in the optical characteristics of the optical system during surgery. In some embodiments, this may include using custom-designed accessories (such as triangular or other geometrically shaped connectors) to mechanically lock the optical components and prevent slippage, rotation, or any other changes, thereby maintaining the optical characteristics required for image calibration and processing.

[0086] 7. A custom light-splitter splits light into spectrally separated beams to acquire multiple sub-images on one or more HSI sensors. Custom light-splitters can be used to split light into spectrally separated beams to acquire multiple sub-images on one or more HSI sensors. In some implementations, microlens arrays or prisms (such as Wollaston prisms) can be used to achieve beam splitting to split a single beam into two or more sub-images on the same sensor. Advantageously, different filters can be used for different beams to acquire hyperspectral images. The use of hyperspectral sensors with multiple spectral response modes can advantageously be combined with such multiple filters to produce responses for different subsets of spectral modes, regardless of the use of a single or multiple sensors.

[0087] 8. High-frequency light source switching to detect and identify ambient light contributions from acquired HSI data. Controlled high-frequency on-and-off switching of light sources can be used to detect and identify ambient light contributions from acquired HSI data. Synchronization between light source switching and HSI camera acquisition can be achieved using a separate triggering mechanism (such as a separate cable connection).

[0088] 9. Suitable heat dissipation measures for surgical setups to ensure low-noise imaging and safe operator handling. Additional heat dissipation measures, including active and passive cooling mechanisms, can be applied to camera systems with sterilization curtains to ensure low-noise imaging and safe operator handling. In some embodiments, this may include providing a heat sink directly attached to the camera. In other embodiments, this may include ensuring thermal conductivity between the camera and the robotic arm or gantry used to assist in positioning the imaging system.

[0089] In addition to connecting the observation instrument to the camera, the eyepiece adapter can also provide control mechanisms for zooming and focusing. Filters can be placed anywhere in the optical path between the origin of the traveling light and its receiving end (such as a camera sensor). In some implementations, filters can be placed in a filter wheel embedded in the light source to limit the light source spectrum according to the camera sensor or clinical requirements.

[0090] The HSI camera can be connected to a computing workstation or equivalent device via a connection that provides power and a high-speed data link suitable for real-time HSI data transmission. The workstation or equivalent device processes the acquired HSI data to visualize the derived information in real time. A sterile surgical drape covering the HSI camera and data cable can be sealed together with a sterile exoscopy lens to ensure the sterility of the overall imaging system.

[0091] Depending on the surgical application, the sterile imaging system can be handheld by the operator or secured to the operating table using a standard robotic arm that allows for control of the imaging system's movement or fixation according to clinical requirements during surgery. In some embodiments where the camera system is sufficiently lightweight, its controlled movement and fixation can be achieved using a sterile or sterile curtained robotic arm attached to a sterile optical observer. This mechanism allows the iHSI system to be positioned at a safe distance outside the surgical cavity, while the eyepiece adapter provides appropriate focusing capability for acquiring HSI data. Other embodiments may include using a robotic positioning arm to hold and control the imaging device.

[0092] The computing workstation 52 performs computational steps to extract information about tissue or object characteristics at the per-pixel level from the acquired low-resolution snapshot hyperspectral imaging data for display during surgery. Figure 11The computing workstation 52 is shown in more detail. It can be seen that the computing workstation 52, which can function as a properly programmed general-purpose computer, includes a processor 1128, a memory 1130, and an input-output interface 1132 from which control input can be obtained from peripheral devices such as a keyboard, foot switch, or pointing device (such as a computer mouse or trackpad). Additionally, another input port 1134 is connected to a hyperspectral imaging camera for capturing hyperspectral imaging data, and an image data output port 1136 is connected to a display for displaying images generated by this embodiment using hyperspectral imaging data as input.

[0093] A computer-readable storage medium 1112 (such as a hard disk, solid-state drive, etc.) is also provided, on which appropriate control software and data are stored to allow operation of embodiments of the invention. Specifically, operating system software 1114 (which provides overall control of the computing system 52) is already stored on the storage medium 1112, and a spatial spectrum demosaicing program 1116 and a parameter estimation program 1118 are also stored on the storage medium. Additionally, a parameter mapping program 1120 is provided. As will be described later, the spatial spectrum demosaicing program 1116 and the parameter estimation program 1118 operate together to provide a first embodiment, while the parameter mapping program operates to combine the functions of the spatial spectrum demosaicing program 1116 and the parameter estimation program 1118 into a single process to provide a second embodiment. The input to both embodiments is in the form of multiple snapshot mosaic images 1126, and the output is various functional or semantic data images 1122, as will be described. Alternatively, an intermediate data structure in the form of a virtual hypercube 1124 may be stored on a computer-readable medium, which is generated during the operation of the first embodiment, as will be described.

[0094] Detailed description of implementation methods related to the HSI method

[0095] In addition to spatial spectral downsampling during snapshot imaging, the hyperspectral information of objects such as tissues is also affected by spatial spectral parasitic effects, leading to the acquisition of hyperspectral images characterized by low spatial and spectral resolution. The disclosed method can perform computational steps that resolve each of these effects independently or jointly in order to obtain per-pixel estimates of tissue or object characteristic information at high spatial resolution.

[0096] Filter response mapping can be used to describe the hyperspectral information of objects with lower spectral resolution acquired by individual band sensors of a snapshot imaging sensor. Other band selection and crosstalk modeling methods can be used to describe the acquired low-spectral and low-spatial-resolution snapshot mosaic images.

[0097] In some implementations (i.e., the first implementation mentioned above), the hyperspectral information of the virtual hypercube, characterized by low spectral density but high spatial resolution, can then be reconstructed by deploying spatial spectral correction methods. Computational parameter estimation methods can be used to infer per-pixel tissue or object characteristics. Example computational methods are disclosed below.

[0098] In other implementations (i.e., the second implementation mentioned above), organizational or object characteristic information can be obtained directly from the acquired low-resolution snapshot data. An example calculation method is disclosed below.

[0099] Those skilled in the art will understand that the direct parameter estimation method can be considered as an inference of tissue characteristic information from a virtual hypercube, wherein the reconstructed virtual hypercube itself corresponds to the estimated tissue characteristic map. (Refer to...) Figure 6 Further details will be provided.

[0100] Figure 6 An overview of the steps involved that can be performed by the computational methods of both the first and second embodiments is presented. In summary, in the first embodiment, at step 62, snapshot mosaic data is captured with low spatial and low spectral resolution. This then undergoes a demosaic process to generate a virtual hypercube 64, which contains virtual high spatial resolution but low spectral resolution data. As described further in detail below, a parameter estimation process can then be performed from the virtual hypercube to obtain the desired high spatial resolution data 66 in the desired parameter space.

[0101] In contrast, in the second embodiment, at 62, snapshot mosaic data is captured again with low spatial and low spectral resolution. Then, a joint demosaicing and parameter estimation process is performed, which, strictly speaking, eliminates the complete generation of the virtual hypercube (although conceptually it may be considered to still generate the parts it needs, even if in practice the computation is performed more directly) (as described in further detail below), in order to obtain the desired high spatial resolution data directly in the desired parameter space.

[0102] More specifically, for real-time hyperspectral imaging, sparse hyperspectral information can be obtained using a snapshot camera system that simultaneously acquires individual spectral bands at different spatial locations in a single shot using a sensor array with a mosaic filter. Such a system can be used in wide-area hyperspectral imaging. and height Effective sensor area Regular grid A snapshot mosaic image is obtained. It should be noted that alternative sensor types with irregular or systematic pixel arrangements similar to mosaic imaging (such as those using microlens arrays) may also be used. Figure 2 The tile-based capture problem can be solved by directly modifying the following method.

[0103] Snapshot Mosaic Describes a typically unknown three-dimensional (3D) high-resolution hypercube The “flattened” two-dimensional (2D) and low-resolution approximations, in which the spectrum Each discrete spectral band provides an approximation of the continuous spectrum within the wavelength range of interest.

[0104] for An example of a mosaic filter can be found in Obtain total from spatial region A single spectral band. For Mosaic, we can assume the filter response operator Its description is from All of the spectrum A mapping of discrete spectral bands, where, typically, This operator will advantageously model the spectral response of the corresponding filter, including the relative radiance of the light source, higher-order harmonics, and spectral leakage of the mosaic filter, but does not include spatial crosstalk across sensor components (which will be considered separately). Assuming for the effective sensor region... Applying this filter band response independently to all individual spatial locations in the hypercube allows for the visualization of the hypercube. via Transformed into a hypercube with lower spectral resolution By assuming the same properties for all mosaic filters, the filter response operator... The extension to the effective sensor area can be achieved using the Kronecker product. Combining the identity operator In terms of form, that is, ,in, Define the band selection operator. It can describe the mapping from a mosaic hypercube to a snapshot mosaic, that is, a three-dimensional... Hypercube to 2D The "flattening" of the mosaic, the two-dimensional mosaic contained in different Individual obtained at a spatial location Spectral information of each band in the spectrum ( Figure 7 This operation can be naturally extended to the entire effective sensor area. Selection operator on Spatial crosstalk between individual adjacent spectral band sensors is caused by the effective sensor area of ​​the sensor. Appropriate crosstalk operator This is used for modeling. As an example, we can assume a kernel size... Convolution is used for each pixel. Crosstalk around the neighborhood is modeled. A spatial crosstalk operator is introduced to address the hybrid sensor response of the neighbor filter. All remaining, primarily spectral, parasitic effects are therefore handled by the spectral filter response operator. This can be addressed by a joint operator. In general, the forward model for acquiring snapshot mosaic images that independently solves the spatial and spectral leakage of the imaging system can be described by this joint operator.

[0105] It will (usually unknown) 3D high-resolution hypercube Mapping to 2D low-resolution snapshot mosaic The following shows how to obtain the true spectral filter response operator for a mosaic sensor. and spatial crosstalk operator A specific example characterizing a hyperspectral snapshot imaging setup. To address differences in the spectrum provided by different light sources, the acquired radiance can be considered. Performing white balance as a preprocessing step, or it can affect the embedding, can be done. middle.

[0106] A filter response operator similar to that of a given mosaic sensor , can be defined Virtual filter response operator for each virtual spectral band The virtual filter response can be selected based on the desired task. Specific choices of the virtual filter response can include a representation of the filter's idealized transfer function, such as the principal response of a Fabry-Pérot optical resonator, which is characteristic of some snapshot imaging systems proposed by Pichette et al., Proc. of SPIE, 2017. To improve interpretability and interoperability, virtual spectral bands can also be selected as regularly spaced spectral bands. Other examples include the spectra of endmembers of particular interest for tissue characterization, such as Hb or HbO2 (…). Figure 4 In practice, virtual filter spectral bands The number can be less than or equal to the filter bands of the mosaic sensor. The quantity. By introducing operators. For spectral correction, it can be ensured that... This is used to establish a spectral mapping from the system's actual filter response to the virtual filter response. The following shows how to obtain the spectral calibration operator. Specific examples ( Figure 8 A). When the spectral response of the filter includes a finite or zero number of spectral parasitic effects (such as higher-order harmonics and spectral leakage), choose... This leads to the description of the identity operator It could be advantageous.

[0107] Using virtual hypercube space Therefore, the definition of "de-mosaicing" or "upsampling" in spatial spectral sensing is derived. This refers to the snapshot image obtained. Reconstructing the Virtual Hypercube ,Right now, This solves the problems of spatial crosstalk and spectral parasitic effects. The demosaicing method is shown below. Specific examples ( Figure 8 B).

[0108] Based on the reconstructed virtual hypercube Parameter estimation methods Used to estimate pixel-level image feature information over the entire effective sensor area, i.e., to estimate features. ,thus, Depends on the property type ( Figure 8 C). For example, it can be... Semantic organization classification used to distinguish between benign and malignant tissue types can... Estimation for pseudo-RGB images.

[0109] Instead of separate demosaicing and parameter extraction steps, i.e. A joint model was proposed. To allow direct analysis of the acquired mosaic image end-to-end method ( Figure 8 D).

[0110] As will be apparent to those skilled in the art, all the computational methods can also be used in multiple camera systems to provide multiplexed video stream data. Furthermore, all the proposed computational methods can lead to different results in areas different from the effective sensor region. Reconstruction on different high-resolution grids. All computational methods can also be based on any other positive loss function besides the norm presented in the examples (e.g., smoothed L1 or squared). Specific assumptions can also be made about the noise level used for error estimation, such as the assumption that the noise is independent but not uniformly distributed across wavelengths.

[0111] White Balance

[0112] The acquired mosaic image captures the radiance from the object. As a preprocessing step, reflectance calculations or white balance can be performed based on the acquired snapshot mosaic image. The reflected signal is calculated using the emissivity in the image. This can be achieved using a separately acquired reference image, which includes the emissivity in the integration time. Acquired dark reference mosaic image and during integration time Acquired white reference mosaic image In some examples, white balance can be achieved by deploying a linear model. That is, in addition to the integral time of the object... The obtained mosaic image It also utilizes the closed shutter to integrate time. and To obtain points for time White reference mosaic image of the reflection patch Dark reference mosaic image and White balance produces a mosaic image of reflections.

[0113] For some examples, the integration time in (2) and They can be the same. In other implementations, This can reduce and avoid potential sensor saturation effects. As those skilled in the art will understand, a white reference can also refer to any means of obtaining a reference that is preferably spectrally neutral. In some embodiments, the use of a gray card can, for example, be combined with similar... The effect of the intensity correction factor is combined to avoid any potential saturation effect when acquiring the white reference. A sterile imaging target with known reflectivity (such as medical equipment available in the operating room) can also be used to estimate the white reference according to (2). An example of such a sterile imaging target could be surgical gauze. Specular reflections from one or more acquired images obtained from various angles and positions relative to the surgical scene can also be used as a substitute for the white reference signal.

[0114] In examples where the imaging setup characteristics are known a priori, white balance can be pre-calculated, thus eliminating the need to obtain white and dark references during intraoperative setup. For various camera methods used for in-operation white balance during the intraoperative use of the imaging system, both white and dark references can be estimated in-plant.

[0115] For fluorescence imaging applications, white balance according to (2) can also be performed. This may include white balance in conjunction with an optical component specifically designed for fluorescence-based imaging, such as an external lens having a sufficient light source and filter for indocyanine green (ICG) or 5-aminolevulinic acid (5-ALA)-induced protoporphyrin IX (PpIX).

[0116] All proposed white balance methods may include temporal processing of the video stream. Such methods can be used to address measurement uncertainties or to capture white balance with spatial variations in non-uniformly reflective targets (such as surgical gauze). Examples may include temporal averaging of a white or dark reference image used in (2).

[0117] Spatial Spectral Calibration

[0118] The true spectral filter response operator can be estimated in a controlled setting. and spatial crosstalk operator (1) Both, to take into account parasitic effects during image acquisition.

[0119] By measuring the characteristics of the sensor in the factory, the measured system filter response operator can be obtained. This can be achieved by using collimated light to acquire snapshot mosaic image data and combining it with imaging targets that have known, typically spatially constant, spectral feature maps to scan across all [the target]. This is achieved using a single wavelength. Combined with (1), spatial spectral calibration of the imaging system can then be performed.

[0120] In linear operators and In the example, let Description Each filter Crosstalk operator for each convolution kernel The unknown parameters are used to simulate the mixing of pixel neighborhood responses for each spectral band. The spatial spectral calibration of the imaging system can be estimated in (1). (In this linear operation mode, it is represented as adopting) (matrix) and (use (This indicates that) both are performed by solving optimization problems such as the following formula.

[0121] Can be used for (3) and / or Apply additional regularization and constraints to the variables, such as positive constraints. In some implementations, it may be advantageous to use a model with the same kernel for all spectral bands. In some examples, further regularization may include using blind source separation methods, which can lead to optimization problems of the following form.

[0122] For a given error threshold and similarity measurement Similarity measures can include normalized mutual information (Kullback-Leibler divergence) or other existing scores. Alternatively, one might focus on minimizing the similarity by a function... The measured deviation from the expected model can be determined, for example, using normality test scores:

[0123] It should be clear that the reformulation of such a constrained optimization model can be accomplished by relaxing the hard constraints by including an additional regularization term in (3).

[0124] Based on (3), it is therefore concluded that, for other examples, this can be achieved by [using spatial regions]. Obtain from a known hypercube Snapshot mosaic image of an object To perform intraoperative spatial spectral calibration of the system, i.e. This is known. This leads to spatial spectral calibration problems of the following form.

[0125] Additional regularization and variable constraints, such as those mentioned above, can be applied.

[0126] In addition to commercially calibrated targets, sterile imaging targets with known reflectivity (such as medical devices available in operating rooms) can also be used to define (6). .

[0127] (3) to (6) The initialization can be based on the measured system filter response operator. To execute this. An advantageous implementation is to perform this initialization using data acquired in the factory, and then to perform refinement using data acquired through point-of-care monitoring.

[0128] The calibration of the optical system during surgical setup (6) can be performed using a device with light transmission characteristics. Different filters are used to achieve this. For example, for areas... The above has a known hypercube representation The target can be aimed at Solve for (6) and finally obtain the snapshot mosaic image. ,thus, This represents point-by-point multiplication in the spectral dimension. Advantageously, this can be achieved by activating a filter wheel in the light source embedded in the imaging system. Switching.

[0129] The calibration of the optical system during surgical setup (6) can also be achieved via temporal processing of the video stream to resolve measurement uncertainties or to enable the use of spatially varying, non-uniformly reflective targets (such as surgical gauze) for white balance. An example could include an acquired snapshot image of a target with known average reflectivity. Time averaging.

[0130] Given virtual filter response Spectral band filtering With virtual hypercube space The mapping between them can be achieved by finding the spectral correction operator. Make To establish this. In the case where linear operators are represented as matrices, this leads to...

[0131] Therefore, additional regularization can be performed.

[0132] It is worth noting that the spatial crosstalk operator is valid if and only if Assumed to be an identity operator And all When the spectral bands are acquired at the same spatial location, the calibration calculation in (7) is simplified to the method described in Pichette et al., Proc. of SPIE, 2017, which is not the case for snapshot mosaic imaging in scenes with varying imaging space. Figure 7 Furthermore, Pichette et al.'s Proc. of SPIE, 2017, did not propose a demosaic step to address both spatial crosstalk and spectral parasitic effects.

[0133] For demosaicing of spatial spectral sensing, conversely, estimating the pseudo-inverse of the spectral correction operator may be useful, i.e.,

[0134] Therefore, regularization can be performed. In other examples, as in (7), an invertible neural network is used as The results of the model can be obtained and .

[0135] As will be apparent to those skilled in the art, all the calibration methods described herein can be performed for multiple camera settings, including different acquisition settings such as different gains.

[0136] Spatial Spectral Sensing Demosaic

[0137] Spatial Spectral Sensing Demosaic Method The aim is to address the parasitic effects present in snapshot imaging, and to improve the performance of the acquired mosaic images. ,Right now, To reconstruct the virtual hypercube .

[0138] A direct and computationally fast method for demosaicing could be to use image resampling (such as linear or cubic resampling) on ​​the calibrated mosaic image, followed by the application of the spectral calibration matrix in (7). In the absence of a model that considers spatial and spectral parasitic effects, this leads to hypercube reconstructions that suffer from fuzziness in both spatial and spectral dimensions, as well as other artifacts such as edge offsets, and thus increase the uncertainty of subsequent tissue characterization. Figure 9 ).

[0139] Using forward model

[0140] Regularized Demosaic Method Based on Inverse Problem (IP) It can be described as

[0141] Using appropriate regularization operators (such as Tikhonov regularization) and constraints on variables (such as positive constraints).

[0142] Based on the virtual filter response The choice of (8) and thus (10) can have very ill-posed properties. Alternative examples of IP-based demosaic may include minimizing the following formula.

[0143] On the contrary, it leads to

[0144] Additional regularization and variable constraints can be applied in (11), including those of the form: Instead Regularization.

[0145] To improve computational efficiency, all operators can be implemented as matrix-free operators.

[0146] If we use Tikhonov regularization for linear modeling and... When norms are used in combination, dedicated linear least squares methods (such as LSMR) can be employed to solve (10) or (11). In the case of regularization based on total variation, the alternating direction method of multipliers (ADMM) can be used. Depending on the operator model, data loss, and the type and combination of regularization terms, other numerical methods, such as primal-dual or front-back separation algorithms, can be used instead.

[0147] To achieve fast computation time for spatial spectral sensing demosaicing used for real-time intraoperative guidance during surgery, machine learning methods can be used, where fast computation time under inference can be achieved at the expense of slower computation time during the training phase. Implementations of machine learning methods can be based on fully convolutional neural networks (CNNs). In some examples, CNNs can be implemented using architectures similar to U-Net.

[0148] In paired samples (i.e., If the database is available, it can be deployed using model parameters. Spatial spectral sensing de-mosaic Supervised (S) machine learning methods. As an example, this can be achieved by minimizing the expected loss function. (i.e., risk or generalized error) to establish optimal parameters. This direct approach can be based on the loss of use. training subset Minimize the empirical risk, that is,

[0149] Using appropriate regularization operators It should be clear that, in addition to methods that increase generalizability, any other loss can be used in (13). This includes stopping optimization when the loss criterion is reached on a separate validation dataset, using data augmentation strategies, dropping data, and so on. In cases where paired datasets are not readily available, such a training dataset can also be constructed using classic IP-based methods. In other implementations, a pair of databases can also be constructed by simulating snapshot data from existing hypercube data via a forward model (9). .

[0150] In other examples, it could be a database. Deployment and utilization of model parameters Unsupervised (U) machine learning methods An example could include finding the optimal parameters. This makes it suitable for training subsets. Self-monitoring methods,

[0151] Using appropriate regularization operators In some implementations, regularization can also be based on cycle consistency loss.

[0152] In other implementations, semi-supervised machine learning methods can be used when there is no exhaustive or sufficient database of representative / real-world paired examples available. A typical implementation of this approach may rely on a formulation combining supervised and unsupervised losses from (13) and (14). Examples may also include using adversarial training methods, such as generative adversarial networks (GANs), to augment the dataset by synthesizing high-fidelity data pairs from the available data.

[0153] Implementations of these examples could include the use of deep neural networks such as CNNs. One example could be based on a single-image super-resolution reconstruction network architecture with a residual network structure, where the initial magnification layer considers regular but spatially shifted hypercube sampling as reflected in mosaic image acquisition. Other approaches could use input layers suitable for irregularly sampled input data, such as layers based on Nadaraya-Watson kernel regression.

[0154] Time consistent with mosaic

[0155] Instead of reconstructing the virtual hypercube from a single mosaic image at a time, a time-consistent approach can be deployed to improve robustness.

[0156] Spatial spectral sensing demosaic can be used for time-consistent virtual hypercube reconstruction between two or more consecutive frames, based on motion compensation between frames.

[0157] Inverse-problem-based methods for temporally consistent spatial spectral sensing demosaicing can be based on optical flow (OF), which is relevant to two consecutive frames. It can be defined as

[0158] Using appropriate regularization operators and In other examples, extensions (15) to multiple frames can be performed.

[0159] Machine learning-based supervised or unsupervised methods for temporally consistent spatial spectral perception demosaicing can be based on video super-resolution methods. These can be based on super-resolution networks with separate or integrated motion compensation, such as optical flow estimation. Other examples can be built on recurrent neural networks (RNNs) such as Long Short-Term Memory (LSTM) networks to process video streams of temporal snapshot image data.

[0160] Similar methods can be used to increase the temporal resolution of visualizations derived from snapshot mosaic imaging. In some examples, this can be achieved by using the displacement obtained during optical flow estimation. (such as (15)), with a half-time step estimate This is to achieve the goal of doubling the frame rate used for HSI data visualization.

[0161] Based on parameter estimation of the virtual hypercube

[0162] Based on the reconstructed virtual hypercube Parameter estimation methods Used to estimate the entire effective sensor area The pixel-level image feature information, i.e., the estimated size. Characteristics .

[0163] The method of spectral non-mixing can be used to estimate the relative abundance of specific end members mixed in the pixel spectrum. For example, for the effective sensor region. The position of each pixel Given a set of reflections or its derived value, Individual staff The spectral mixing can be described by a linear spectral mixing model:

[0164] in, Indicates random error. Indicates pixel position end staff The relative abundance ratio. This is determined by defining the end-member matrix. and local abundance Model (16) can be written as .exist and In the case of spectral non-mixing, the inverse problem-based method can be read as

[0165] Utilize appropriate regularization And variable constraints, such as positivity.

[0166] In one example, the regularization in (17) can be omitted, which results in the relative abundance being calculated directly using the normal equation. .

[0167] The following can be used: (17) and Other options for measuring the difference between them include cosine distance. Specific assumptions about the noise level in (16) can also be made, such as that the noise is independent but not uniformly distributed across wavelengths.

[0168] Other methods can be based on supervision methods similar to the demosaicing methods (13) and (14), respectively. or unsupervised methods .

[0169] Examples of spectral non-mixing can include end-member oxyhemoglobin (HbO2) and deoxyhemoglobin (Hb) per pixel. Figure 4 The spectral characteristics of their molar extinction coefficients are shown to be unmixed. This is used to estimate the relative abundance of the correlation. A simple model can be based on a reflective hypercube. Absorption rate estimation In addition to considering the end-members of scattering loss, the derived abundances of oxyhemoglobin and deoxyhemoglobin can also be considered. Used to estimate total hemoglobin (or blood perfusion). and oxygen saturation ( Figure 10 a and Figure 10 b).

[0170] Other examples of spectral non-mixing can include tissue differentiation based on known spectral reflectance feature maps of tissue type.

[0171] Virtual hypercubes that preserve NIR reflectance information can also be used to prevent spectral mixing during fluorescence imaging based on the known absorption and emission spectra of fluorescent compounds such as PpIX or ICG. This can also be used for quantitative fluorescence imaging to estimate the concentration of fluorescent compounds.

[0172] Pseudo-RGB images can be obtained, for example, from a virtual hypercube using the CIE RGB color matching function. Figure 10c). If the virtual hypercube does not present a spectral band covering the visible spectrum used for RGB reconstruction, or if the virtual hypercube only partially covers the spectral band, a colorization / color regression method can be deployed. Particularly in the case of NIR imaging, this can include supervised or unsupervised methods for colorization to estimate per-pixel RGB information. One example could include using a recurrent adversarial network for unpaired samples of surgical RGB images and virtual hypercube reconstruction. Other methods can be based on the higher-order response of the optical system used in the visible range. In one implementation using an NIR imaging sensor, the higher-order response (generally considered as an unwanted spectral response that needs to be eliminated outside the effective range of the sensor) can be specifically used to acquire spectral measurements outside the NIR region ( Figure 3 By utilizing the known filter response curves of a sensor across the spectrum, signals covering the RGB color information used for image reconstruction can be sequentially acquired in the NIR or visible range using switching between filters. In some examples, this switching can be advantageously achieved by using a filter wheel embedded in the light source.

[0173] Other examples of parameter estimation can include segmenting tissue or surgical tools using data-driven supervised, semi-supervised, or unsupervised / self-supervised machine learning methods.

[0174] In other examples, the virtual hypercube and its per-pixel reflectance information can be used to estimate optical tissue properties. One example could include absorption coefficients, which can be estimated using methods similar to fitting parametric models for inverse addition-doubling (IAD) or inverse Monte Carlo methods. Based on absorption estimates obtained using, for example, IAD, parametric model regression, supervised, or semi-supervised machine learning methods can be designed to estimate absorption maps from the virtual hypercube.

[0175] Joint segmentation and parameter estimation methods can be used. One example could include automatic tissue segmentation, which can be used to provide tissue-specific scattering before obtaining a more accurate estimate of the absorption coefficient.

[0176] Another example could include automated segmentation of tissue or surgical tools for more robust tissue parameter estimation. This could include addressing the suppression of image artifacts (such as specular reflections) or contributions from non-tissue-related signals.

[0177] Other image analysis methods can be used to extract information relevant to surgical decisions from the virtual hypercube.

[0178] Parameter estimation from snapshot imaging

[0179] As mentioned above, in the second embodiment, it can also be achieved through a calculation method. From the acquired mosaic image Parameter extraction is performed directly.

[0180] Such a model can be based on an "ideal" parameter mapping. Prior knowledge.

[0181] Similar to introducing a spectral correction operator between spectral band filtering and the virtual hypercube space. The concept can be used to establish models with parameters. mapping , making This mapping can be determined by the following formula:

[0182] Utilize appropriate regularization And variable constraints, such as positivity. Utilizing (11), that is,

[0183] Using forms The appropriate regularization term leads to

[0184] In other examples, there are model parameters. pseudo-reversal It can be determined by the following formula:

[0185] Utilize appropriate regularization And variable constraints, such as positivity. The forward model can then be defined as

[0186] Similar to the examples (10) to (14) mentioned above, parameter mappings can be estimated using inverse problem-based methods, supervised, semi-supervised, and unsupervised methods. .

[0187] It should be noted that invertible models, such as invertible neural networks, can be used instead of estimation. or This invertible model not only learns the forward mapping but also builds the corresponding inverse at the same time.

[0188] As an example, this can be derived from a spectral unmixing model used to estimate the abundance of oxyhemoglobin and deoxyhemoglobin. By transforming the linear spectral mixing model (16) from space... Expand to The normal equations lead to the following explicit formulation:

[0189] thus, .

[0190] In other examples, data-driven machine learning methods can be used to obtain pseudo-RGB images from snapshot images. One example could include using a recurrent adversarial network to process unpaired samples of surgical RGB images and snapshot mosaic images.

[0191] Uncertainty estimation

[0192] Although the proposed forward model is usually well-defined, the problem of estimating the inverse is often undefined, such as pixel-by-pixel reconstruction of tissue characteristic parameters obtained from low-resolution snapshot images. Clearly, for all proposed computational methods, additional uncertainty quantification capabilities can be introduced to estimate the uncertainty of the obtained outcome. This can include methods such as random dropout sampling, probabilistic inference, estimator ensembles, or increased test time.

[0193] In other implementations, invertible mappings can be used in presented models, such as those obtained through invertible neural networks, which not only learn the forward mappings but also establish the corresponding inverse processes. This approach can also be used to recover a complete posterior distribution capable of capturing uncertainties in the obtained solution estimates.

[0194] Uncertainty estimates can be displayed to users or used as part of a computational pipeline or visualization strategy. As an example, uncertainty estimates related to parameter estimation can be used to select and display only estimates that meet a given determinism criterion, such as a threshold.

[0195] Detailed description of implementation methods related to the iHSI system

[0196] We first present an exemplary embodiment of the system according to the invention, which integrates two state-of-the-art industrial HSI cameras as part of an iHSI system setup. The two iHSI embodiments are then evaluated and scored according to the proposed design requirements (Tables 1 and 2). Following this, we perform controlled chessboard experiments to demonstrate that reliable reflectance measurements can be obtained using these embodiments employing two HSI cameras. Ex vivo experiments demonstrate reflectance characteristics across a range of tissue types. In this embodiment, the standard tripod system for photography advantageously allows for a versatile imaging configuration in a controlled environment. Finally, we describe a successful, ethically approved clinical feasibility case study of hospitalized patients demonstrating that the real-time iHSI embodiment can be seamlessly integrated into the surgical workflow while adhering to clinical requirements in the operating room, such as aseptic technique.

[0197] HSI system implementation method.

[0198] As part of the proposed iHSI invention for specific implementation, two hyperspectral imaging cameras were investigated (Table 3): (i) a line scan HSI implementation using an Imec fast scan VNIR (i.e., visible (VIS) to near-infrared (NIR) region) camera, and (ii) a snapshot HSI implementation using a Photonfocus MV0-D2048x1088-CO1-HS02-160-G2 camera.

[0199] The line scan implementation captures hypercube images at a spatial resolution of up to 3650 × 2048 pixels for 150+ spectral bands between 470 nm and 900 nm. The imaging speed for the entire hypercube varies between 2 s and 40 s, depending on the acquisition parameters, illumination, and imaging target. The camera, without optics, measures 10 × 7 × 6.5 cm. 3 It measures in size and weighs 0.58 kg. Line scanning technology is characterized by high SNR across the spectral range. An integrated shutter automatically measures dark current, so only a white reference image needs to be manually acquired for image calibration.

[0200] The Photonfocus camera deploys an Imec Snapshot Mosaic CMV2K-SM5x5-NIR sensor, which acquires 25 spectral bands in a 5×5 mosaic pattern within the 665 nm to 975 nm spectral range. Utilizing the sensor resolution of 2048×1088 pixels, hyperspectral data is acquired at a spatial resolution of 409×217 pixels per spectral band. Video rate imaging of the snapshot data is achieved at speeds up to 50 FPS, depending on the acquisition parameters. The camera, without optics, measures 3×3×5.4 cm. 3 Its dimensions and weight are 0.08 kg.

[0201] The passive prototype cooling system is manufactured with rounded edges and mounted by two heat sinks on the side of the camera to maintain a low operating temperature during image acquisition and thus low imaging noise. Figure 12 a). This increases the overall dimensions in all directions by approximately 3 cm, with an additional weight of approximately 0.2 kg.

[0202] The Asahi Spectra MAX-350 light source (300 W xenon lamp) is used to provide broadband light. Based on experiments, either a VIS module or a UV-NIR mirror module is available, providing light in the 385 nm to 740 nm and 250 nm to 1050 nm regions, respectively. When using the UV-NIR mirror module, an additional 400 nm long-pass filter (Asahi Spectra XUL0400) is placed in front of the mirror module to suppress ultraviolet (UV) light, thereby improving light safety distribution. For the Photonfocus camera, a 670 nm long-pass filter (Asahi Spectra XVL0670) is placed in the filter wheel to avoid signal contamination caused by out-of-band sensor response originating from the sensor's sensitivity to light in the VIS spectrum during image acquisition. The light intensity on the Asahi light source can be adjusted in integer increments between 5% and 100%. The light source was connected to the Karl Storz 0° VITOM surgical endoscope 20916025AA via a Karl Storz 495NCS fiber optic cable, allowing imaging at a safe distance between 25 cm and 75 cm. A custom adapter was used to insert the light guide into the Asahi light source. The endoscope was attached to the corresponding HSI camera via a separate RVA Synergies C-mount 18-35 mm ZOOM endoscope connector, which also provides manual zoom and focus mechanisms.

[0203] For calibration during all experiments, a 95% reflectance patch was used to obtain a white reference image. For the Photonfocus camera, a separate dark reference image was obtained using a lens with a cover.

[0204] Verify the implementation method of iHSI by comparing it with the design specifications.

[0205] Based on the design requirements specified in Table 2, the applicability of both line scan and snapshot camera-based iHSI implementations to the intraoperative setup was evaluated. Table 3 provides a summary of the evaluation.

[0206] Starting with system requirements, a combination of sterile curtains and aseptic components can be used to ensure the asepticity of both camera setups (T2). However, it is evident from the corresponding camera specifications that the snapshot camera allows for a more compact iHSI system given its smaller camera size and weight (T2, T3). The xenon light source uses a UV-NIR mirror module (250 nm to 1050 nm) to provide sufficient energy in both the VIS and NIR spectral ranges (T7), such as... Figure 13 As shown.

[0207] Optical safety is advantageously ensured by using a 400 nm long-pass filter to block UV light (T8). The light source allows for remote configuration using a serial communication protocol, thereby enabling the adjustment of filter wheel position and light intensity using customized software (T10). Similarly, both line scan and snapshot imaging systems are supplied with an API interface to allow for remote control and software integration.

[0208] Device manipulation is crucial to ensuring the safe installation and movement of the imaging system during surgery without adversely affecting surgical workflow and aseptic technique (T2, T3, T9, T10). Due to the compactness of snapshot camera-based systems, this can be easily achieved using a robotic arm construction (T9). However, for line scan camera-based systems, weight and form factor do not allow for the same approach. Installation and pivoting of an imaging system in a rotating position, supported solely by the endoscope adapter and robotic arm, is considered unsafe.

[0209] Both camera setups rely on the same optical setup and adapter and allow imaging at safe distances from the surgical cavity ranging from 250 mm to 750 mm (T12). When using a fixed circular 50 mm FOV at a working distance of 250 mm, the depth of field for both systems is 35 mm (T12 to T14) based on the exoscope manufacturer’s specifications

[33] . Manual focusing and zoom adjustments are possible using the endoscope adapter to provide clear imaging at a given focal length (T11). In terms of HSI data quality, the spatial and spectral image resolution of the line scan camera is far superior to that of the snapshot camera (T15, T17). In particular, in addition to the fewer spectral bands sampled by the snapshot camera, additional post-processing methods such as demosaicing are required to address the sparse spatial sampling in order to obtain HSI data information with a sufficiently high spatial resolution for tissue analysis greater than 409 × 217 pixels per spectral band (T15, T17). While line scan systems cover a wide spectral range in both the VIS and NIR regions to allow for rich feature extraction, snapshot cameras only provide NIR spectral information. Furthermore, line scan technology is supplied along with high-fidelity HSI signal measurements with a high signal-to-noise ratio. In contrast, signals acquired using snapshot imaging are characterized by multi-peak spectral banding and crosstalk contamination from mosaic imaging sensors (which needs to be addressed). Thus, line scan systems can potentially extract a wider range of relevant surgical features. However, the acquisition speed of line scan cameras, ranging from 2 s to 40 s per image, can disrupt surgical workflows without providing the video rate information required for real-time surgical guidance (T19). In particular, motion artifacts are prone to occur when imaging non-static targets. In contrast, the high frame rate of up to 50 FPS of snapshot cameras allows for easy capture of real-time visualization of moving targets (T19). For line scan cameras, image calibration can be achieved by acquiring a white reference image solely due to its integrated shutter. For snapshot cameras, both dark and white reference images need to be acquired (T18). For both camera setups, robust calibration methods that can handle varying lighting and imaging scenarios are crucial for estimating reliable HSI information for intraoperative surgical guidance.

[0210] In general, in these implementations, line scan cameras offer superior imaging quality compared to snapshot cameras. However, given their form factor, more complex mounting mechanisms advantageously ensure safe and aseptic manipulation of the camera during surgery. Furthermore, their relatively low imaging rate does not allow for HSI data capture without disrupting the surgical workflow, crucial for providing real-time information for seamless surgical guidance. Nevertheless, their imaging characteristics still ensure high-quality HSI in controlled settings. In contrast, video-rate snapshot cameras allow for compact and aseptic iHSI systems that can be integrated into surgical workflows using standard clinical robotic arm constructions. For reliable tissue analysis, image processing methods can advantageously address the reduced spatial and spectral image resolution, in addition to the lower signal quality inherent in mosaic snapshot sensors.

[0211] Chessboard Study: Verification of iHSI System Implementation Methods.

[0212] The proposed intraoperative optics system implementation (i.e., endoscope adapter and exoscope) was used to test line scan and snapshot cameras to acquire HSI data in a controlled experiment using a datacolor SpyderCHECKR checkerboard supplied with 48 color patches. For this experiment, the Asahi light source was used with a UV-NIR module combined with a 400 nm long-pass filter to provide light from 400 nm to 1050 nm. A reference spectrum was obtained using an OceanOptics Maya 2000Pro 200-1100 nm spectrometer with an OceanOptics QR600-7-VIS125BX reflective probe. Figure 4 ).

[0213] For the line scan camera, an exposure time of 10 ms and a gain of 1.2 were used to acquire images. For the snapshot camera, an exposure time of 15 ms and a gain of 2 were used to acquire images. Spectral-calibrated hypercube reflectance data was provided using proprietary software for image analysis of both camera systems using the default image calibration files provided by the cameras. Specifically, no dedicated system-wide calibration was performed during image calibration to account for the specific light source intensity spectrum. Figure 13 ) and individual optical components of the iHSI system (such as filters, endoscope adapters, and exoscopes).

[0214] Both the line scan camera and the snapshot camera were positioned 35 cm from the chessboard, thereby acquiring images individually for each color patch. For each calibrated hypercube image, five circular regions with a radius of 10 pixels were manually segmented on the color patch for spectral analysis. Figure 14 c).

[0215] Figure 15 A comparison is provided between reference data and spectral information obtained by the iHSI system using line scan and snapshot cameras. It can be seen that the estimated reflectance for both the line scan and snapshot iHSI systems primarily follows the spectrometer reference measurements.

[0216] In vitro studies: Calf cadaver experiments.

[0217] In vitro experiments using fresh calf cadavers were performed in a controlled environment to study tissue characteristics of both line scan and snapshot camera iHSI implementations performed at Balgrist University Hospital, Zurich, Switzerland. Calf cadavers were chosen because their anatomy is similar to that of the human spine.

[36]

[0218] Various tissue types are exposed for tissue analysis, including tendons, muscles, bones, joint capsules, dura mater, and spinal cord. To achieve optimal orientation and positioning for imaging cadaver tissue samples, a standard tripod system is advantageously used to mount the iHSI camera system. Figure 16 For both line scan and snapshot cameras, a secure attachment is achieved using a custom adapter plate with 1 / 4-20 UNC and 3 / 8-16 UNC threaded holes. Figure 12 (b) An additional Thorlabs DCC3260C RGB camera was used in the experiment to provide high-resolution 1936×1216 RGB imaging. The camera without optics measures 2.9×3.5×4.4cm. 3 It measures 0.04 kg in size and weighs 0.04 kg. Given its C-mount camera lens mount, it can be used with the same endoscope adapter as part of the same iHSI setup. Additionally, its housing is supplied with 1 / 4-20 UNC threaded holes suitable for attaching a quick-release tripod plate.

[0219] Imaging of the exposed tissue using three cameras follows Figure 17The procedure outlined in the document describes how the use of individual, distinguishable references across the VIS and NIR spectra ensured retrospective alignment of images acquired using different cameras. A set of six needles (colored red, black, blue, white, green, and yellow) tied with nylon thread facilitated manipulation during experiments. After positioning the first HSI camera (line scan or snapshot camera) and adjusting zoom and focus to image the tissue sample, references were placed on the tissue to ensure they were within the field of view (FOV). The references were then removed from the scene for image capture and carefully reinserted before acquiring a second image using the same HSI camera to avoid anatomical changes. Without disturbing the scene, the HSI camera was swapped with an RGB camera using a tripod quick-release mechanism to acquire an RGB image of the tissue sample with the references. Subsequently, without altering the scene, the RGB camera was swapped with a second HSI camera on the tripod. Minor adjustments to camera position, zoom, and focus are typically required before image acquisition to ensure the target tissue is in focus and the references are within the FOV. After carefully removing the reference object, another image of the same scene was acquired without altering the settings. For spectral analysis, the neurosurgeon manually annotated the relevant tissue types in the pseudo-RGB line scan image, which was obtained by assigning the red, green, and blue channels to wavelengths of 660 nm, 570 nm, and 500 nm, respectively. Alignment between all images was achieved using affine point-based registration by manually annotating the circular reference object

[37] . The manual segmentation in the line scan image space was then propagated to the snapshot image space for analysis using the obtained point-based affine registration.

[0220] During the ex vivo experiments, only the VIS mirror module was available as the light source, thus providing light between 385 nm and 740 nm. For NIR imaging using the snapshot camera, an additional 670 nm long-pass filter in the filter wheel of the light source was activated. For all scenes, the snapshot camera used a gain of 3.01 and an exposure time of 20 ms, thereby performing video imaging to acquire multiple images of each individual static scene. On average, this resulted in acquiring 18 snapshot mosaic images per scene, the mean image of which was used for spectral analysis. For the line scan camera, a gain of 2 and an exposure time of 20 ms were used. The light intensities were set to 100%, 100%, and 50% for the snapshot camera, line scan camera, and high-resolution RGB camera, respectively. Imaging of all cameras was performed with indoor lights off and blackout curtains drawn to reduce the influence of background light. To simplify the imaging workflow, reference data for image calibration was acquired once for the line scan camera and once for the snapshot camera at the beginning and end of the experiment. Therefore, the same white balance information from each HSI camera was used for data calibration of all images associated with different anatomical locations.

[0221] Figure 18 Provided for Figure 16 The comparison in section b of estimated reflectance curves between 470 nm and 740 nm from line scans and snapshot-based iHSI systems for eight different anatomical scenes. For snapshot cameras, only 5 of the 23 reconstructed spectral bands were available for analysis of measurements between 670 nm and 740 nm. Generally, for overlapping spectral bands between cameras, the relative distribution and qualitative behavior of reflectance values ​​across tissue types are well aligned.

[0222] In terms of clinical feasibility case studies for patients: spinal fusion surgery.

[0223] After evaluating proposed iHSI implementations using both line-scan and snapshot cameras in light of the critical design requirements for the surgery, combining quantitative and qualitative assessments, the snapshot-based implementation is advantageous in providing real-time HSI that can be seamlessly integrated into the surgical workflow. To confirm this hypothesis, we conducted an intraoperative clinical feasibility case study at Balgrist University Hospital, Zurich, Switzerland, as part of a spinal fusion surgery. This study was approved by the Cantonal Ethics Committee (BASEC Nr: req-2019-00939).

[0224] Figure 5A schematic diagram of an iHSI implementation deployed during surgery is presented. In addition to the system components described previously, a standard Karl Storz robotic arm is used to securely attach the iHSI camera system to the operating table via a gripper (28272UGK) and an articulated L-shaped frame (28272HC). Secure attachment of the snapshot HSI camera to the VITOM exoscope is achieved via a suitable rotary socket (28172HR) and clamping cylinder (28272CN). Overall sterility of the system is ensured by autoclaving the robotic arm, exoscope, and light guide before surgery, and by covering the camera and associated cables with a sterile curtain.

[0225] The primary objective of the intraoperative clinical feasibility case study was to confirm the system's integration into the standard surgical workflow. To focus on the objective, we chose to mimic current optical camera systems during surgery, using white light between 385 nm and 740 nm without a 670 nm long-pass filter. For the snapshot camera, a gain of 4 and an exposure time of 20 ms were selected, with the light source providing 100% light intensity. A laptop running customized software for real-time interaction with the camera system and for data visualization was placed on a trolley at a safe distance outside the sterile environment. A monitor connected to the operating room provided a live display of the captured video rate HSI data. Figure 19 a). Specifically, this allows for immediate feedback and interaction with the surgical team to adjust camera position and orientation in addition to endoscope adapter settings, thereby acquiring focus alignment data for the surgical area of ​​interest. Using this device, in vivo imaging is performed at eight different stages during surgery to acquire HSI data for various tissue types, including skin, adipose tissue, scar tissue, fascia, muscle, bone, pedicle screws, and dura mater (…). Figure 19 b). Imaging of individual anatomical structures lasts from 6 to 44 seconds with minimal disruption to the surgical workflow. Following a successful surgery that utilizes a seamless transition to acquire HSI data, a final recording of 3 minutes and 16 seconds is performed to capture imaging data covering the surgical cavity.

[0226] Discussion and Conclusion

[0227] Previous work has highlighted the potential of HSI as a non-contact, non-ionizing, non-invasive, and label-free imaging modality for intraoperative tissue characterization. Despite a large body of research exploring the clinical potential of HSI in surgery, to our knowledge, no HSI system has been developed that adheres to stringent clinical requirements, including aseptic technique and seamless integration into surgical workflows that can provide real-time information for intraoperative surgical guidance.

[0228] Here, we present an implementation of an HSI system suitable for intraoperative surgical guidance in open surgery. However, our invention allows for adaptation to endoscopic and microscopic surgeries. We implement our invention in two state-of-the-art industrial HSI camera systems based on line-scan or snapshot technologies and evaluate their suitability for surgical use. Based on our established criteria, we propose an intraoperative HSI implementation and perform scoring against these requirements by considering two HSI cameras. We perform controlled checkerboard experiments to demonstrate that reliable reflectance measurements can be obtained using the proposed implementation employing two HSI cameras. Ex vivo experiments are performed to investigate the reflectance properties of a range of tissue types, including tendons, muscles, bones, joint capsules, dura mater, and spinal cord, with the two iHSI camera setup mounted on a standard tripod system that allows for a universal imaging configuration in a controlled environment. In particular, this has proven to be a suitable setup for line-scan cameras to provide high-resolution data in both the spatial and spectral dimensions of the VIS and NIR spectra for use in ex vivo tissue analysis. The iHSI system allows for a seamless and safe transition throughout all stages of spinal fusion surgery and acquires video-rate HSI data for multiple tissue types, including skin, adipose tissue, fascia, muscle, bone, pedicle screws, and dura mater. Our successful clinical feasibility case studies demonstrate that the proposed iHSI system integrates seamlessly into the surgical workflow by respecting key clinical requirements such as aseptic technique and provides wide-field-of-view video-rate HSI imaging. By developing a data-driven information processing pipeline, such video-rate HSI data can be utilized to provide real-time, wide-field-of-view tissue characterization for intraoperative surgical guidance.

[0229] The proposed in vitro setup can be used to combine experiments to obtain high-resolution line scans and low-resolution snapshot HSI data. This can advantageously provide key information for developing real-time demosaicing and tissue differentiation methods for snapshot HSI.

[0230] As part of an exemplary implementation, any small camera that meets the camera size and weight requirements outlined in Table 1 can be directly integrated into the proposed iHSI system setup. It is also advantageous to enable HSI acquisition with variable working distance, field of view, depth of field, and depth of focus (Table 2), which in turn will enable the device to be integrated with a variety of commercial endoscopic surgical systems

[34] and will ensure compliance with current microsurgical standards

[38] .

[0231] Our in vivo clinical feasibility studies demonstrated that the implementation of our invention integrates well into standard surgical workflows and is capable of capturing HSI data. Overall, surgical and operating room team members found the tested implementation simple to use, despite the need for routine training to ensure smooth operation during surgery. The tested implementation did not pose any safety concerns for team members, and the system's size, weight, and portability were acceptable in maintaining a smooth surgical workflow.

[0232] Table 1: An overview of the advantageous functional design requirements for implementations of hyperspectral imaging systems for intraoperative surgical guidance. The corresponding advantageous technical requirements in Table 2 are listed in the rightmost column.

[0233]

[0234] Table 2: Overview of advantageous technical requirements for implementations of hyperspectral imaging systems for intraoperative surgical guidance. The corresponding advantageous functional requirements in Table 1 are listed in the rightmost column.

[0235]

[0236] Table 3: Validation of the intraoperative hyperspectral imaging implementation based on whether the advantageous requirements outlined in Table 2 are met. An evaluation was performed using the exemplary implementation described herein on two camera setups with ratings (R) of 0 (minimum advantageous requirements not met), 1 (minimum advantageous requirements met), and 2 (target advantageous requirements met).

[0237]

[0238] Those skilled in the art will understand that various further modifications (whether by adding, deleting, or replacing) to the above examples are intended to provide additional implementations, any and all of which are intended to be covered by the appended claims.

[0239] References

[0240] [1] IJ Gerard, M. Kersten-Oertel, K. Petrecca, D. Sirhan, JAHall, and DL Collins, “Brain shift in neuronavigation of brain tumors: Areview,” Medical Image Analysis, vol. 35, pp. 403-420, jan 2017. [Online].Available: http: / / dx.doi.org / 10.1016 / j.media.2016.08.007 https: / / linkinghub.elsevier.com / retrieve / pii / S1361841516301566

[0241] [2] P. A. Helm, R. Teichman, S. L. Hartmann, and D. Simon, “SpinalNavigation and Imaging: History, Trends, and Future,” IEEE Transactions on Medical Imaging , vol. 34, no. 8, pp. 1738-1746, aug 2015. [Online].Available: http: / / ieeexplore.ieee.org / document / 7008544 /

[0242] [3] G. Lu and B. Fei, “Medical hyperspectral imaging: a review,” Journal of Biomedical Optics , vol. 19, no. 1, p. 010901, jan 2014. [Online].Available: http: / / biomedicaloptics.spiedigitallibrary.org / article.aspx?doi=10.1117 / 1.JBO.19.1.010901

[0243] [4] M. Halicek, H. Fabelo, S. Ortega, G. M. Callico, and B. Fei, “In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques:Revealing the Invisible Features of Cancer,” Cancers, vol. 11, no. 6, p. 756,may 2019. [Online]. Available: https: / / www.mdpi.com / 2072-6694 / 11 / 6 / 756

[0244] [5] J. Shapey, Y. Xie, E. Nabavi, R. Bradford, S. R. Saeed, S.Ourselin, and T. Vercauteren, “Intraoperative multispectral and hyperspectrallabel-free imaging: A systematic review of in vivo clinical studies,” Journal of Biophotonics , vol. 12, no. 9, p. e201800455, sep 2019. [Online].Available: https: / / onlinelibrary.wiley.com / doi / abs / 10.1002 / jbio.201800455

[0245] [6] N. T. Clancy, G. Jones, L. Maier-Hein, D. S. Elson, and D.Stoyanov, “Surgical spectral imaging,” Medical Image Analysis , vol. 63, p.101699, 2020. [Online]. Available: https: / / doi.org / 10.1016 / j.media.2020.101699

[0246] [7] J. H. G. M. Klaessens, M. Nelisse, R. M. Verdaasdonk, and H. J.Noordmans, “Non-contact tissue perfusion and oxygenation imaging using a LEDbased multispectral and a thermal imaging system, first results of clinicalintervention studies,” in Advanced Biomedical and Clinical Diagnostic Systems XI , A. Mahadevan-Jansen, T. Vo-Dinh, and W. S. Grundfest, Eds., vol. 8572,mar 2013, p. 857207. [Online]. Available: http: / / proceedings.spiedigitallibrary.org / proceeding.aspx?doi=10.1117 / 12.2003807

[0247] [8] M. Mori, T. Chiba, A. Nakamizo, R. Kumashiro, M. Murata, T.Akahoshi, M. Tomikawa, Y. Kikkawa, K. Yoshimoto, M. Mizoguchi, T. Sasaki, andM. Hashizume, “Intraoperative visualization of cerebral oxygenation usinghyperspectral image data: a two-dimensional mapping method,” International Journal of Computer Assisted Radiology and Surgery , vol. 9, no. 6, pp. 1059-1072, nov 2014. [Online]. Available: http: / / link.springer.com / 10.1007 / s11548-014-0989-9

[0248] [9] E. Kho, L. L. de Boer, K. K. Van de Vijver, F. van Duijnhoven,M.-J. T. Vrancken Peeters, H. J. Sterenborg, and T. J. Ruers, “HyperspectralImaging for Resection Margin Assessment during Cancer Surgery,” Clinical Cancer Research, vol. 25, no. 12, pp. 3572-3580, Jun 2019. [Online].Available: http: / / clincancerres.aacrjournals.org / lookup / doi / 10.1158 / 1078-0432.CCR-18-2089

[0249]

[10] H. Fabelo, S. Ortega, R. Lazcano, D. Madroñal, GM Callicó, E.Juárez, R. Salvador, D. Bulters, H. Bulstrode, A. Szolna, J. Piñeiro, C.Sosa, AJ O'Shanahan, S. Bisshopp, M. Hernández, J. Morera, D. Ravi, B.Kiran, A. Vega, A. Báez-Quevedo, G.-Z. Yang, B. Stanciulescu, and R. Sarmiento, “An Intraoperative Visualization System Using HyperspectralImaging to Aid in Brain Tumor Delineation,” Sensors , vol. 18, no. 2, p. 430, Feb 2018. [Online]. Available: http: / / www.mdpi.com / 1424-8220 / 18 / 2 / 430

[0250]

[11] N. Chiang, JK Jain, J. Sleigh, and T. Vasudevan, “Evaluation of hyperspectral imaging technology in patients with peripheral vascular disease,” Journal of Vascular Surgery, vol. 66, no. 4, pp. 1192-1201, oct2017. [Online]. Available: http: / / dx.doi.org / 10.1016 / j.jvs.2017.02.047https: / / linkinghub.elsevier.com / retrieve / pii / S0741521417309308

[0251]

[12] M. Desjardins, J.-P. Sylvestre, R. Jafari, S. Kulasekara, K.Rose, R. Trussart, J. D. Arbour, C. Hudson, and F. Lesage, “Preliminaryinvestigation of multispectral retinal tissue oximetry mapping using ahyperspectral retinal camera,” Experimental Eye Research , vol. 146, pp. 330-340, may 2016. [Online]. Available: https: / / linkinghub.elsevier.com / retrieve / pii / S0014483516300653

[0252]

[13] L. C. Cancio, A. I. Batchinsky, J. R. Mansfield, S. Panasyuk, K.Hetz, D. Martini, B. S. Jordan, B. Tracey, and J. E. Freeman, “HyperspectralImaging: A New Approach to the Diagnosis of Hemorrhagic Shock,” The Journal of Trauma: Injury, Infection, and Critical Care , vol. 60, no. 5, pp. 1087-1095, may 2006. [Online]. Available: http: / / journals.lww.com / 00005373-200605000-00025

[0253]

[14] L. Khaodhiar, T. Dinh, K. T. Schomacker, S. V. Panasyuk, J. E.Freeman, R. Lew, T. Vo, A. A. Panasyuk, C. Lima, J. M. Giurini, T. E. Lyons,and A. Veves, “The Use of Medical Hyperspectral Technology to EvaluateMicrocirculatory Changes in Diabetic Foot Ulcers and to Predict ClinicalOutcomes,” Diabetes Care , vol. 30, no. 4, pp. 903-910, apr 2007. [Online].Available: http: / / care.diabetesjournals.org / cgi / doi / 10.2337 / dc06-2209

[0254]

[15] B. Fei, G. Lu, X. Wang, H. Zhang, J. V. Little, M. R. Patel, C.C. Griffith, M. W. El-Diery, and A. Y. Chen, “Label-free reflectancehyperspectral imaging for tumor margin assessment: a pilot study on surgicalspecimens of cancer patients,” Journal of Biomedical Optics , vol. 22, no. 08,p. 1, aug 2017. [Online]. Available: https: / / www.spiedigitallibrary.org / journals / journal-of-biomedical-optics / volume-22 / issue-08 / 086009 / Label-free-reflectance-hyperspectral-imaging-for-tumor-margin-assessment / 10.1117 / 1.JBO.22.8.086009.full

[0255]

[16] H. Fabelo, M. Halicek, S. Ortega, M. Shahedi, A. Szolna, J. Piñeiro, C. Sosa, A. O'Shanahan, S. Bisshopp, C. Espino, M. Marquez, M. Hernandez, D. Carrera, J. Morera, G. Callico, R. Sarmiento, and B. Fei, “DeepLearning-Based Framework for In Vivo Identification of Glioblastoma Tumorusing Hyperspectral Images of Human Brain,” . Sensors , vol. 19, no. 4, p. 920,Feb 2019. [Online]. Available: http: / / www.mdpi.com / 1424-8220 / 19 / 4 / 920

[0256]

[17] Gioux S, Mazhar A, Lee BT, Lin SJ, Tobias AM, DJCuccia, Stockdale A, Oketokoun R, Ashitate Y, Kelly E, Weinmann M, Durr NJ, Moffitt LA Moffitt, Durkin AJ, Tromberg BJ, and Frangioni JV imaging system,” Journal of Biomedical Optics , vol. 16, no. 8, p. 086015, 2011. [Online]. Available: http: / / biomedicaloptics.spiedigitallibrary.org / article.aspx?doi=10.1117 / 1.3614566

[0257]

[18] S. L. Best, A. Thapa, N. Jackson, E. Olweny, M. Holzer, S. Park,E. Wehner, K. Zuzak, and J. A. Cadeddu, “Renal Oxygenation Measurement DuringPartial Nephrectomy Using Hyperspectral Imaging May Predict AcutePostoperative Renal Function,” Journal of Endourology , vol. 27, no. 8, pp.1037-1040, aug 2013. [Online]. Available: http: / / www.liebertpub.com / doi / 10.1089 / end.2012.0683

[0258]

[19] H. J. Noordmans, C. Ferrier, R. de Roode, F. Leijten, P. vanRijen, P. Gosselaar, J. Klaessens, and R. Verdaasdonk, “Imaging the seizureduring surgery with a hyperspectral camera,” Epilepsia , vol. 54, no. 11, pp.e150-e154, nov 2013. [Online]. Available: http: / / doi.wiley.com / 10.1111 / epi.12386

[0259]

[20] M. Barberio, F. Longo, C. Fiorillo, B. Seeliger, P. Mascagni, V.Agnus, V. Lindner, B. Geny, A.-L. Charles, I. Gockel, M. Worreth, A. Saadi,J. Marescaux, and M. Diana, “HYPerspectral Enhanced Reality (HYPER): aphysiology-based surgical guidance tool,” Surgical Endoscopy, vol. 34, no. 4,pp. 1736-1744, apr 2020. [Online]. Available: https: / / doi.org / 10.1007 / s00464-019-06959-9 http: / / link.springer.com / 10.1007 / s00464-019-06959-9

[0260]

[21] J. Yoon, J. Joseph, D. J. Waterhouse, A. S. Luthman, G. S. D.Gordon, M. di Pietro, W. Januszewicz, R. C. Fitzgerald, and S. E. Bohndiek,“A clinically translatable hyperspectral endoscopy (HySE) system for imagingthe gastrointestinal tract,” Nature Communications , vol. 10, no. 1, p. 1902,dec 2019. [Online]. Available: http: / / dx.doi.org / 10.1038 / s41467-019-09484-4http: / / www.nature.com / articles / s41467-019-09484-4

[0261]

[22] R. Mühle, H. Ernst, S. B. Sobottka, and U. Morgenstern,“Workflow and hardware for intraoperative hyperspectral data acquisition inneurosurgery,” Biomedical Engineering / Biomedizinische Technik , jul 2020.[Online]. Available: https: / / www.degruyter.com / view / journals / bmte / ahead-of-print / article-10.1515-bmt-2019-0333 / article-10.1515-bmt-2019-0333.xml

[0262]

[23] A. Kulcke, A. Holmer, P. Wahl, F. Siemers, T. Wild, and G.Daeschlein, “A compact hyperspectral camera for measurement of perfusionparameters in medicine,” Biomedical Engineering / Biomedizinische Technik ,vol. 63, no. 5, pp. 519-527, oct 2018. [Online].Available: http: / / www.degruyter.com / view / j / bmte.2018.63.issue-5 / bmt-2017-0145 / bmt-2017-0145.xml

[0263]

[24] H. Köhler, A. Kulcke, M. Maktabi, Y. Moulla, B. Jansen-Winkeln,M. Barberio, M. Diana, I. Gockel, T. Neumuth, and C. Chalopin, “Laparoscopicsystem for simultaneous high-resolution video and rapid hyperspectral imagingin the visible and near-infrared spectral range,” Journal of Biomedical Optics , vol. 25, no. 08, aug 2020. [Online]. Available: https: / / www.spiedigitallibrary.org / journals / journal-of-biomedical-optics / volume-25 / issue-08 / 086004 / Laparoscopic-system-for-simultaneous-high-resolution-video-and-rapid-hyperspectral / 10.1117 / 1.JBO.25.8.086004.full

[0264]

[25] J. Pichette, A. Laurence, L. Angulo, F. Lesage, A. Bouthillier,D. K. Nguyen, and F. Leblond, “Intraoperative video-rate hemodynamic responseassessment in human cortex using snapshot hyperspectral optical imaging,” Neurophotonics , vol. 3, no. 4, p. 045003, oct 2016. [Online]. Available:http: / / neurophotonics.spiedigitallibrary.org / article.aspx?doi=10.1117 / 1.NPh.3.4.045003

[0265]

[26] M. Ewerlöf, M. Larsson, and E. G. Salerud, “Spatial and temporalskin blood volume and saturation estimation using a multispectral snapshotimaging camera,” in Imaging, Manipulation, and Analysis of Biomolecules, Cells, and Tissues XV , D. L. Farkas, D. V. Nicolau, and R. C. Leif, Eds.,vol. 10068, feb 2017, p. 1006814. [Online]. Available: http: / / proceedings.spiedigitallibrary.org / proceeding.aspx?doi=10.1117 / 12.2251928

[0266]

[27] K. Dijkstra, J. van de Loosdrecht, L. R. B. Schomaker, and M. A.Wiering, “Hyperspectral demosaicking and crosstalk correction using deeplearning,” Machine Vision and Applications, vol. 30, no. 1, pp. 1-21, feb2019. [Online]. Available: https: / / doi.org / 10.1007 / s00138-018-0965-4 http: / / link.springer.com / 10.1007 / s00138-018-0965-4

[0267]

[28] G. Tsagkatakis, M. Bloemen, B. Geelen, M. Jayapala, and P.Tsakalides, “Graph and Rank Regularized Matrix Recovery for Snapshot SpectralImage Demosaicing,” IEEE Transactions on Computational Imaging , vol. 5, no.2, pp. 301-316, jun 2019. [Online]. Available: https: / / ieeexplore.ieee.org / document / 8584100 /

[0268]

[29] J. Shapey, Y. Xie, E. Nabavi, D. Ravi, S. R. Saeed, R. Bradford,S. Ourselin, and T. Vercauteren, “Towards intraoperative hyperspectralimaging: design considerations for neurosurgical applications,” Proceedings of Hamlyn Symposium on Medical Robotics , p. 18, 2018.

[0269]

[30] L. Ricciardi, K. L. Chaichana, A. Cardia, V. Stifano, Z.Rossini, A. Olivi, and C. L. Sturiale, “The Exoscope in Neurosurgery: AnInnovative “Point of View”. A Systematic Review of the Technical, Surgical,and Educational Aspects,” World Neurosurgery , vol. 124, pp. 136-144, apr2019. [Online]. Available: https: / / doi.org / 10.1016 / j.wneu.2018.12.202https: / / linkinghub.elsevier.com / retrieve / pii / S1878875019300804

[0270]

[31] P. G. Yock, S. Zenios, J. Makower, T. J. Brinton, U. N. Kumar,F. T. J. Watkins, L. Denend, T. M. Krummel, and C. Q. Kurihara, Biodesign: The process of innovating medical technologies , 2nd ed. Cambridge UniversityPress, 2015.

[0271]

[32] S. H. Yun and S. J. J. Kwok, “Light in diagnosis, therapy andsurgery,” Nature Biomedical Engineering , vol. 1, no. 1, p. 0008, jan 2017.[Online]. Available: http: / / www.nature.com / articles / s41551-016-0008

[0272]

[33] K. Nishiyama, “From Exoscope into the Next Generation,” Journal of Korean Neurosurgical Society , vol. 60, no. 3, pp. 289-293, may 2017.[Online]. Available: http: / / jkns.or.kr / journal / view.php?doi=10.3340 / jkns.2017.0202.003

[0273]

[34] D. J. Langer, T. G. White, M. Schulder, J. A. Boockvar, M.Labib, and M. T. Lawton, “Advances in Intraoperative Optics: A Brief Reviewof Current Exoscope Platforms,” Operative Neurosurgery , vol. 19, no. 1, pp.84-93, jul 2020. [Online]. Available: https: / / academic.oup.com / ons / article / 19 / 1 / 84 / 5569799

[0274]

[35] S. Thorpe, D. Fize, and C. Marlot, “Speed of processing in thehuman visual system,” Nature , vol. 381, no. 6582, pp. 520-522, jun 1996.[Online]. Available: http: / / www.nature.com / articles / 381520a0

[0275]

[36] P. C. Cotterill, J. P. Kostuik, G. D’Angelo, G. R. Fernie, andB. E. Maki, “An anatomical comparison of the human and bovine thoracolumbarspine,” Journal of Orthopaedic Research , vol. 4, no. 3, pp. 298-303, 1986.[Online]. Available: http: / / doi.wiley.com / 10.1002 / jor.1100040306

[0276]

[37] A. Myronenko and X. Song, “Point Set Registration: CoherentPoint Drift,” IEEE Transactions on Pattern Analysis and Machine Intelligence ,vol. 32, no. 12, pp. 2262-2275, dec 2010. [Online]. Available: http: / / arxiv.org / abs / 0905.2635%0Ahttp: / / dx.doi.org / 10.1109 / TPAMI.2010.46 http: / / ieeexplore.ieee.org / document / 5432191 /

[0277]

[38] J. Schwiegerling, C. Chancy, and R. C. Dimalanta, “Depth ofFocus Measurement of an Ophthalmic Surgical Microscope,” Investigative Ophthalmology & Visual Science , vol. 56, no. 7, p. 1925, 2015.

[0278]

[39] L. Ayala, S. Seidlitz, A. Vemuri, S. J. Wirkert, T. Kirchner, T.J. Adler, C. Engels, D. Teber, and L. Maier-Hein, “Light source calibrationfor multispectral imaging in surgery,” International Journal of Computer Assisted Radiology and Surgery , vol. 15, no. 7, pp. 1117-1125, jul 2020.[Online]. Available: https: / / doi.org / 10.1007 / s11548-020-02195-y http: / / link.springer.com / 10.1007 / s11548-020-02195-y

Claims

1. A method for determining parameters of a desired target image from hyperspectral imagery, the method comprising the following steps: A hyperspectral image sensor is used to capture hyperspectral snapshot mosaic images of a scene, the snapshot mosaic images having relatively low spatial resolution and low spectral resolution; Spatial-spectral-aware demosaicing is performed on the snapshot mosaic image to generate a virtual hypercube of snapshot mosaic image data. The spatial-spectral-aware demosaicing includes an algorithm that optimizes the cost function by considering spatial crosstalk between neighboring pixels corresponding to different spectral bands in the snapshot mosaic image. The virtual hypercube includes image data with relatively high spatial resolution compared to the snapshot mosaic image. From the image data in the virtual hypercube, determine the relatively high spatial resolution parameters of the desired target image; as well as The determined relatively high spatial resolution parameters are output as a representation of the desired target image.

2. The method according to claim 1, wherein, The demosaicing process involves resampling the snapshot mosaic image and then applying a spectral calibration matrix.

3. The method according to claim 1, wherein, The demosaic process includes machine learning.

4. The method according to any one of claims 1 to 3, wherein, Based on motion compensation between frames, the demosaic is temporally consistent across two or more consecutive frames.

5. The method according to any one of claims 1 to 3, further comprising the step of: A white balance operation is performed on the hyperspectral image sensor before capturing the hyperspectral snapshot mosaic image.

6. The method according to claim 5, wherein, The white balance operation includes: acquiring a reference image separately, the reference image being included during integration time. Dark reference mosaic image and during integration time White reference mosaic image ; and deploying linear models, where, in addition to the objects in integral time The obtained mosaic image It also utilizes the closed shutter to integrate time. and To obtain points for time White reference mosaic image of the reflection patch Dark reference mosaic image and And the white balance operation produces by The given reflection mosaic image.

7. The method according to any one of claims 1 to 3, further comprising the step of: Before capturing the hyperspectral snapshot mosaic image, a spatial spectral calibration operation is performed on the hyperspectral image sensor.

8. The method according to claim 7, wherein, Operator for estimating the true spectral filter response in a controlled setting and spatial crosstalk operator To account for parasitic effects during image acquisition.

9. The method of claim 8, further comprising the step of: By using collimated light to acquire snapshot mosaic image data and combining it with imaging targets that have known, typically spatially constant, spectral feature maps, all [the data is] scanned. The characteristics of the hyperspectral image sensor are measured at several wavelengths to obtain the measured system filter response operator. .

10. The method according to any one of claims 1 to 3, wherein, The step of determining the relatively high spatial resolution parameter further includes: analyzing pixel-level hyperspectral information to obtain its unique end-member composition characterized by specific spectral feature maps.

11. The method according to any one of claims 1 to 3, wherein, The step of determining the relatively high spatial resolution parameter further includes estimating the tissue characteristics at each spatial location based on the reflectance information from hyperspectral imaging.

12. A method for determining parameters of a desired target image from hyperspectral imagery, the method comprising the following steps: A hyperspectral image sensor is used to capture hyperspectral snapshot mosaic images of a scene, the snapshot mosaic images having relatively low spatial resolution and low spectral resolution; Joint demosaicing and parameter estimation are performed from the snapshot mosaic image to determine the relatively high spatial resolution parameters of the desired target image; as well as The determined relatively high spatial resolution parameters are output as a representation of the desired target image. The demosaicing is spatially-spectrally aware and includes an algorithm that optimizes the cost function by taking into account spatial crosstalk between neighboring pixels corresponding to different spectral bands in the snapshot mosaic image.

13. A system for determining parameters of a desired target image from hyperspectral imagery, the system comprising: A hyperspectral image sensor, configured to capture hyperspectral images of a scene; processor; as well as A computer-readable storage medium storing computer-readable instructions that, when executed by the processor, cause the processor to control the system to perform the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing a computer program that, when executed, causes a hyperspectral imaging system to perform the method according to any one of claims 1 to 12.

Images