Method and system for performing mosaic removal and spectral feature estimation simultaneously
Patent Information
- Application Number
- JP2022575173
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-02-19
- Filing Date
- 2021-05-26
- Publication Date
- 2026-09-30
- Estimated Expiration
- 2041-05-26
Smart Images
Figure 0007926919000042 
Figure 0007926919000043 
Figure 0007926919000044
Abstract
Description
[Technical Field]
[0001] Embodiments of the present invention generally relate to image processing and video processing, and more particularly to a system and method for acquiring hyperspectral images acquired in real time, which in some embodiments are acquired in a medical context, and for performing image processing on such images. [Background technology]
[0002] Many difficult decisions made during surgery, which can potentially change a patient's life, still rely on the surgeon's subjective visual assessment. This is partly because, even with the latest advanced surgical techniques, it may still be impossible to reliably identify critical structures during surgery. The need for more detailed, non-qualitative, wide-field visualization and tissue characterization during surgery is becoming increasingly apparent across various surgical specialties.
[0003] As a first example, in neuro-oncology, surgical intervention (gross total resection, GTR) is often the primary treatment, aiming to safely remove as much abnormal tissue as possible. Performing GTR requires balancing the risk of postoperative morbidity associated with damaging delicate areas responsible for vital biological functions, such as nerves and blood vessels. During surgery, preoperative information (e.g., MRI or CT) of the patient's anatomical tissue on the operating table can be mapped using navigation solutions, such as those disclosed in U.S. Patent No. 9,788,906(B2). However, navigation based on preoperative imaging does not account for changes during surgery. Surgeons can visualize the tissue being operated on by using interventional imaging and detection, such as surgical microscopy, fluorescence imaging, point-based Raman spectroscopy, ultrasound, and intraoperative MRI, either alone or in conjunction with navigation information. However, current intraoperative imaging-based tissue identification remains difficult due to severe surgical constraints in the clinical setting (e.g., intraoperative MRI or CT) or inaccurate tumor depiction (e.g., ultrasound or fluorescence imaging). Fluorescence-guided surgery using 5-aminolevulinic acid (5-ALA)-induced protoporphyrin IX (PpIX) is increasingly being used in neurotumor surgery. Other fields, including bladder cancer, are also benefiting from PpIX fluorescence-guided surgery. However, visualization of malignant tissue boundaries is ambiguous due to the accumulation of tumor markers in healthy tissue, a lack of quantitative accuracy partly due to confounding effects between time-dependent fluorescence and tissue autofluorescence, side effects, and applicability only to specific tumors, as reviewed in Suero Molia et al., Neurosurgical Review, 2019. There is a wealth of prior art aimed at better identifying neurosurgical tissues, and these prior arts clearly demonstrate that better imaging during surgery is a good means of improving outcomes for patients undergoing these challenging surgeries.
[0004] As a second example, necrotizing enterocolitis (NEC) is a devastating neonatal disease that often requires surgical treatment and has the potential for serious side effects. NEC is characterized by ischemic necrosis of the intestinal mucosa, leading to neonatal perforation, widespread peritonitis, and, in severe cases, death. Three out of every 1,000 newborns develop NEC, 85% of cases occur in infants with very low birth weight (<1500g), and 30% die despite state-of-the-art treatment, according to a review by Hull et al., Journal of the American College of Surgeons, 2014. Surgical treatment for NEC includes primary peritoneal drainage, diagnostic and confirmatory surgery, and / or laparotomy with bowel resection. A major challenge for surgeons performing NEC laparotomy is determining the amount of intestine to resect, given the long-term risk of the infant developing short bowel syndrome. This involves weighing the infant's chances of recovery against leaving inadequately perfused intestine in place. Currently, there are no standardized imaging guidance techniques for NEC laparotomy. Therefore, surgical planning for resection relies on the surgeon's judgment, dexterity, and perceptual abilities. If in doubt, a rough tissue incision can be made to assess bleeding. The mortality rate for NEC is thought to be potentially reduced through early diagnosis, better monitoring, and improved surgical treatment.
[0005] As discussed in Shapey et al., Journal of Biophotonics, 2019, multispectral imaging and hyperspectral imaging (hereinafter collectively referred to as hyperspectral imaging (HSI)) are new optical imaging techniques with the potential to transform surgical procedures. However, it is unclear whether current systems can provide real-time, high-resolution tissue characterization for surgical guidance. HSI is a safe, non-contact, non-ionizing, and non-invasive optical imaging modality with attractive properties for surgical use. By splitting light into multiple spectral bands far beyond the visible range, HSI reveals nuanced information about tissue characteristics, surpassing conventional color information and allowing for more objective tissue characterization. In HSI, within a given time frame, the collected data spans a three-dimensional space composed of two spatial dimensions and one spectral dimension. Each of these three-dimensional frames is commonly referred to as a hyperspectral image or hypercube. The concept of using HSI for medical applications is known and has been studied for decades, as described in U.S. Patent No. 6,937,885(B1). Classically, HSI relied on scanning the space and / or spectrum to acquire a complete hypercube. Because scanning was time-consuming, these methods could not provide a live display of the hyperspectral image. Recently, compact sensors called snapshot HSIs have been developed that can acquire HSI data in real time. Such snapshot sensors acquire hyperspectral images at a video rate that can achieve approximately 30 or more hyperspectral frames per second, typically at the expense of both spectral and spatial resolution.Instead of acquiring a high-density hypercube, i.e., fully sampled spectral information (in the z direction) at each spatial pixel of a given scene (xy plane), a snapshot hyperspectral camera acquires a subsampled hyperspectral image in a single shot. This acquisition typically uses a tiled or mosaic pattern, as detailed in Pichette et al., Proc. of SPIE, 2017.
[0006] Here, we define a hyperspectral imaging system as real-time if it can acquire images at a video rate suitable for providing a live display of hyperspectral imaging information on the order of tens of frames per second.
[0007] As shown in Shapey et al., Journal of Biophotonics, 2019, and further explained below from the perspective of prior art, while current HSI systems can acquire important information during surgery, they do not currently provide a means of delivering wide-field, real-time information with sufficient resolution to support surgical guidance.
[0008] Hyperspectral imaging for medical applications is described by various acquisition principles. The main one relies on sequential filtering of light at the detector. An early example is U.S. Patent No. 5,539,517(A), which proposes an interferometer-based method in which a predetermined set of linear combinations of spectral intensities is sequentially acquired by scanning. U.S. Patent No. 6,937,885(B1), from the same period, proposes the sequential acquisition of HSI data with the help of tunable filters such as Liquid Crystal Tuneable Filters (LCTFs), combined with prior knowledge of expected tissue responses, in order to acquire data according to a given diagnostic protocol. U.S. Patent No. 8,320,996(B2) proposes an improvement over programmable spectral separators such as LCTFs to sequentially acquire spectral bands, extract information relevant to a specific diagnostic protocol, and project a summarized pseudo-color image onto the region of interest within the image area. U.S. Patent Application Publication No. 2851662(A2) describes sequential acquisition of spectral imaging information using a slit-shaped aperture combined with a dispersive element and mechanical scanning. Because these methods rely on sequential acquisition, they are not suitable for real-time wide-field imaging. Furthermore, none of these studies present a means to improve the resolution of the acquired HSI.
[0009] In addition to filtering light at the detection end, the use of filtered excitation light has also been explored in medical HSIs. As a first example, U.S. Patent Publication No. 2013 / 0245455(A1) proposed an HSI with multiple LED light sources that are switched on in a specific order to sequentially acquire multiple spectral bands. Following a similar approach, International Publication No. 2015 / 135058(A1) presented an HSI system that requires optical communication between a remote light source and a spectral filtering device to scan a series of illumination filters. As with their corresponding detection filtering, these systems are not suitable for real-time imaging and do not offer solutions for improving HSI resolution.
[0010] In the medical field, HSI data sources are still being integrated into more complex configurations, some of which are being considered to provide pathology-related identification information. U.S. Patent Application Publication 2016 / 0278678(A1) relies on the projection of spatially modulated light for depth-resolved fluorescence imaging combined with hyperspectral imaging. U.S. Patent No. 10,292,771(B2) discloses a surgical imaging system that potentially includes an HSI device and utilizes a specific surgical port by performing a procedure to reduce its reflectivity. U.S. Patent No. 9,788,906(B2) uses hyperspectral imaging as a source of information that can be used to detect the phase of a medical procedure and configure the imaging device accordingly. HSI-derived tissue classification is disclosed in European Patent Application Publication 3545491(A1), where clustering assigns the same classification to all pixels belonging to the same cluster. Tissue classification based on HSI data is also available in International Publication 2018 / 059659(A1) brochure. Although there is potential interest in their use during surgery, none of these imaging systems offer means of acquiring real-time HSI or improving the resolution of HSI images, nor do they offer means of producing high-resolution tissue characterization of classification maps.
[0011] HSI has been studied for the evaluation of a variety of clinical conditions, including peripheral vascular disease
[11] , retinal eye disease
[12] , hemorrhagic shock
[13] , healing of foot ulcers in diabetic patients
[14] , and cancer detection
[15] , but its use in vivo during surgery has been limited to only a few clinical research cases[5]. For example, the HELICoiD research system
[10] has demonstrated promising clinical results
[16] for in vivo brain tumor detection, but its size is contraindicated for clinical use during surgery. Other systems have further demonstrated the potential of intraoperative HSI (iHSI), but these systems have been presented for the evaluation of tissue perfusion and oxygenation during surgery, including breast
[17] , oral cancer[7], kidney
[18] , epilepsy
[19] , neurovascular[8], and gastrointestinal surgery[20, 21]. However, these tend to generate motion artifacts due to insufficient imaging speed of the dynamic scene during surgery. Recently, two intraoperative systems based on pushbroom HSI cameras have been published that could be integrated into surgical workflows. One
[22] involves attaching a pushbroom HSI system
[23] to a surgical microscope to acquire in vivo neurosurgical data, while the other
[24] presents a laparoscopic HSI camera that has been tested during esophageal surgery. Although these systems have shown potential to support surgical workflows, their limited imaging speed is likely to remain a deterrent to their adoption during surgery.
[0012] To increase real-time imaging speeds, as mentioned in the above section on image acquisition, recently developed snapshot HSI camera systems are being used for evaluating cerebral perfusion in neurosurgery
[25] and preclinical cutaneous perfusion analysis
[26] . However, while snapshot HSI sensors enable real-time HSI acquisition through video-rate imaging, their spatial resolution is limited and must be addressed in a post-processing step called demosaicking [27,28]. Furthermore, previously presented snapshot iHSI papers are not methodologically detailed and do not address important design considerations for ensuring seamless integration into surgical workflows.
[0013] While various HSI systems have been tested in surgical settings and the potential of iHSI has been studied, to the best of our knowledge, no HSI system has been presented that satisfies stringent clinical requirements, including the ability to maintain sterility and ensure seamless integration into the surgical workflow, i.e., the ability to provide real-time information for intraoperative surgical guidance.
[0014] Outside the medical field, sensors capable of acquiring HSI data in real time have recently been proposed. European Patent Application Publication No. 3348974(A1) presents a hyperspectral mosaic sensor that interleaves spectral filters at the pixel level to generate real-time HSI data that is spatially and spectrally sparse. Many aberrations are expected in such sensors. Pichette et al., Proc. of SPIE, 2017, presented a calibration approach that can correct some of the distortion of the observed spectrum, but did not present a method to improve spatial resolution. Dijkstra et al., Machine Vision and Applications, 2019, presents a learning-based approach to snapshot HSI acquired by a mosaic sensor. In particular, a hypercube reconstruction approach is presented. This approach focuses solely on demosaiking / crosstalk correction for hypercube reconstruction. However, it does not disclose how to extract parameters of the desired target image from the hyperspectral image. While this paper discusses the effects of spectral crosstalk and the sparse nature of the sensor, it does not model or directly capture the combined effects of various distortions. It simplifies by assumptions and separates crosstalk correction from upscaling. Alternative methods for acquiring snapshot HSI data have been proposed, such as coded aperture snapshot spectral imaging (CASSI) presented in Wagadarikar et al., Applied Optics, 2008.These imaging systems typically involve numerous optical elements, such as dispersed optics, coded apertures, and several n-lenses, often resulting in impractical form factors for surgical use. Similar to mosaic sensors, CASSI systems present not only difficult trade-offs between temporal, spectral, and spatial resolution, but also complex and computationally expensive reconstruction techniques. U.S. Patent Application Publication 2019 / 0096049(A1) proposed reconstructing CASSI-based HSI data using a learning-based method combined with an optimization method. While the computational complexity is reduced and the system can acquire raw data in real time, means for performing real-time reconstruction are not disclosed. While sensors such as mosaic sensors and CASSI may be usable in surgery, there is a need to demonstrate how to integrate these sensors into a real-time system capable of displaying high-resolution HSI-derived images and simultaneously providing maps, such as identifying images for tissue characterization or classification to support surgery.
[0015] Prior art demonstrates that the problem of tissue characterization during surgery arises in many surgical fields, and that this has been addressed in various ways. Hyperspectral imaging shows great potential in this field. However, to the best of our knowledge, no method has been disclosed that can provide wide-field, high-resolution tissue-related information derived in real time from hyperspectral imaging during surgery. Therefore, there is a need for systems and methods that enable improved real-time resolution of hyperspectral imaging and the characterization of related tissues. [Overview of the Initiative]
[0016] Embodiments of the present invention provide a method and system for determining parameters of a desired target image from a hyperspectral image of a scene. The parameters can represent various aspects of the imaged scene, particularly its physical properties. For example, in some medical imaging processes (contexts), the properties being imaged may be pixel-by-pixel blood perfusion or oxygenation saturation level information. In one embodiment, the parameters are obtained by collecting hyperspectral images with lower spectral and lower spatial resolution, and then constructing a virtual hypercube of information with higher spatial resolution using a spatial spectrum-aware demosaicing process. This virtual hypercube is used to estimate the desired parameters at higher spatial resolution. Alternatively, in another embodiment, instead of performing estimation after constructing the virtual hypercube, demosaicing and parameter estimation calculations are performed together to obtain high spatial resolution parameters directly from the lower spectral and spatial resolution hyperspectral images. Various white level and spectral calibration calculations can also be performed to improve the obtained results.
[0017] With regard to the development of systems for hyperspectral imaging, the inventors have made four contributions: (i) In contrast to previous research
[29] , the inventors have systematically addressed a set of design requirements, including functional and technical requirements, which are essential for an iHSI system that provides real-time wide-field HSI information for seamless surgical guidance in a highly constrained operating room (OR); (ii) The inventors present and evaluate a series of iHSI embodiments to these requirements by examining two state-of-the-art industrial HSI camera systems based on line scan and snapshot imaging techniques; (iii) The inventors conduct ex vivo animal tissue experiments in a controlled environment using exemplary iHSI embodiments to study tissue properties using both camera systems; and (iv) The inventors report the use of an iHSI embodiment (Figure 5) in real time during a case study of clinical feasibility in an ethically approved patient as part of spinal fusion surgery, thus successfully verifying the inventors' assumption that the invention can be seamlessly integrated into the OR without interrupting the surgical workflow.
[0018] In view of the above, according to the first embodiment, a method for determining parameters of a desired target image from a hyperspectral image, comprising: capturing a hyperspectral snapshot mosaic image of a scene using a hyperspectral image sensor, wherein the snapshot mosaic image has relatively low spatial resolution and spectral resolution; performing mosaic removal on the snapshot mosaic image to generate a virtual hypercube of snapshot mosaic image data, wherein the virtual hypercube includes image data having a relatively higher spatial resolution than the snapshot mosaic image; determining relatively high spatial resolution parameters of a desired target image from the image data of the virtual hypercube; and outputting the determined relatively high resolution parameters as a representation of the desired target image.
[0019] In one example, said demosaicing is spatially spectrum-aware. For example, demosaicing may comprise image resampling, such as linear or cubic resampling of a snapshot mosaic image, followed by application of a spectral calibration matrix. In another example, demosaicing may comprise machine learning.
[0020] Additionally or alternatively, said demosaicing may be temporally aligned between two or more consecutive frames based on inter-frame motion compensation.
[0021] A further example further comprises the step of performing a white balance adjustment operation on a hyperspectral image sensor before capturing said hyperspectral snapshot mosaic image. In one example, the white balance adjustment operation may comprise the step of separately acquiring reference images, the step of acquiring a dark reference mosaic image and a white reference mosaic image respectively at integration times τ d and τ w and the step of executing a linear model, wherein the obtained mosaic image w of an object at integration time τ τ in addition, the white reference mosaic image and the dark reference mosaic image of the reference tile at integration time τ w are obtained with the shutter closed at integration times τ and τ w , and the white balance adjustment operation may generate a reflection mosaic image given by the following formula. [Math.]
[0022] In a further example, before capturing said hyperspectral snapshot mosaic image, a spatial-spectral calibration operation is performed on the hyperspectral image sensor. During the calibration operation, an actual spectral filter response operator and a spatial crosstalk operator T:W→W represented by the following formula are estimated in a controlled setting to address parasitic effects during image acquisition. [Math.]
[0023] In addition, a further example is the step of measuring the characteristics of the hyperspectral image sensor to obtain a measured system filter response operator, wherein snapshot mosaic image data is acquired using collimated light, and all n associated with a known, typically spatially constant spectral feature imaging target. Λ The process may further include sweeping the wavelength.
number
[0024] In one example, the step of determining relatively high spatial parameters further includes analyzing pixel-level hyperspectral information for the composition of intrinsic edge components characterized by specific spectral features.
[0025] In one example, the step of determining relatively high spatial parameters further includes the step of estimating tissue characteristics for each spatial location (typically a pixel) from reflectance information of hyperspectral imaging, such as pixel-level tissue absorption information.
[0026] Another example of the present disclosure is a method for determining parameters of a desired target image from a hyperspectral image, comprising the steps of: capturing a hyperspectral snapshot mosaic image of a scene using a hyperspectral image sensor; performing mosaic removal and parameter estimation from the snapshot mosaic image, where the snapshot mosaic image has relatively low spatial resolution and relatively low spectral resolution, in order to determine relatively high spatial resolution parameters of a desired target image; and outputting the determined relatively high resolution parameters as a representation of the desired target image. In this additional example, all of the above-described white balance adjustment and calibration calculations may be employed.
[0027] Further aspects of the present disclosure provide a system for hyperspectral imaging of a target region. The system includes a light source for illuminating the target region, a hyperspectral image sensor configured to capture one or more hyperspectral images of the target region, and an optical scope connected to the hyperspectral image sensor, wherein the image of the target region generated by the optical scope during use is acquired by the hyperspectral image sensor.
[0028] In one example, the system according to this disclosure is a system for hyperspectral imaging of a target region. The system includes a light source for illuminating the target region and at least one hyperspectral image sensor configured to capture a hyperspectral image of the target region, wherein the system is configured to acquire a plurality of hyperspectral subimages of the target region on at least one image sensor.
[0029] In one example, the system according to the present disclosure is a system for hyperspectral imaging of a target region. The system is a system for hyperspectral imaging of a target region and includes a light source for illuminating the target region and a hyperspectral image sensor configured to capture a hyperspectral image of the target region, wherein the system is configured to control the switching of the light source at a predetermined frequency.
[0030] In a further example, the system according to the present disclosure is a system for hyperspectral imaging of a target region. The system includes a light source for illuminating the target region and a hyperspectral image sensor configured to capture a hyperspectral image of the target region, the system including heat dissipation means connected to the hyperspectral image sensor.
[0031] The system in any of the above examples may be further configured to determine parameters of a target region from a hyperspectral image. The system further includes a processor and a computer-readable storage medium for storing computer-readable instructions, which, when executed by the processor, cause the processor to control the system to perform the methods described in this section.
[0032] Furthermore, another example provides a computer-readable storage medium for storing a computer program. When executed, the computer program causes a hyperspectral imaging system to perform one of the methods illustrated in this section, according to any of the embodiments described above. Further features and aspects of the present invention will become apparent from the appended claims. [Brief explanation of the drawing]
[0033] Further features and advantages of the present invention will become apparent from the following description of embodiments and are presented by reference only to the examples and drawings, where similar reference numerals refer to similar parts. [Figure 1] This display shows a typical arrangement of filters within a mosaic sensor. [Figure 2] This is a schematic representation of the mosaic sensor array that constitutes the active sensor area of a snapshot mosaic imaging system. [Figure 3] This graph shows an example of the response of a near-infrared 5x5 mosaic sensor. [Figure 4] This is an indication of the molar extinction coefficients of oxyhemoglobin (HbO2) and deoxyhemoglobin (Hb). [Figure 5] This figure illustrates an example of a sterilization imaging system that can be used for real-time hyperspectral imaging. [Figure 6] This is a commutative diagram representing the steps performed by the computation method during spatial spectrum calibration, virtual hypercube reconstruction, and parameter estimation. [Figure 7]This display provides a rough comparison between the sparse hyperspectral information acquired by 2D snapshot mosaic imaging and the information captured by a 3D hypercube. [Figure 8] This diagram illustrates the procedure for spatial spectrum calibration, mosaic removal considering the spatial spectrum, and parameter estimation using a virtual hypercube or acquired snapshot imaging data. [Figure 9] This illustrates the shortcomings of mosaic removal methods that do not recognize spatial spectra. [Figure 10] This display shows different tissue characteristic parameter maps extracted from a virtual hypercube. [Figure 11] This is a diagram of a computer system relating to one embodiment of the present invention. [Figure 12] This is a schematic diagram of an embodiment of line scan imaging and snap scan imaging. [Figure 13] This is a schematic diagram of a xenon light source with and without a UV filter. [Figure 14] This displays the setup for the checkerboard experiment. [Figure 15] This graph shows an example of a reconstructed checkerboard experiment. [Figure 16] This displays the settings for an in vitro experiment. [Figure 17] This is a display of an example camera acquisition sequence during in vitro imaging to capture HSI data of the spinal cord and root canals. [Figure 18] This displays a comparison of estimated reflectance curves. [Figure 19] This image shows the intraoperative HSI setting (iHSI) during a spinal fusion study, using an example of an in vivo snapshot mosaic image. [Modes for carrying out the invention]
[0034] In one aspect, embodiments described herein relate to a computer-based method and computer system for obtaining a hyperspectral image from lower-resolution mosaic image data acquired in real time in order to determine image characteristic information representing several physical properties of an imaged sample. This method may include the following: • Acquisition of hyperspectral imaging data using medical devices suitable for use in sterile environments. • Application of a data-driven computational model to provide hyperspectral information with a resolution strictly higher than any spectral band of the original data.
[0035] The imaging system used for data acquisition in this method and system may consist of one or more hyperspectral imaging (HSI) cameras and one or more light stimuli provided by light sources. Its applications can be combined with scopes such as exoscopy or endoscopes, which are provided as part of the optical path of the imaging system.
[0036] The imaging system may be handheld, for example, fixed to the operating table with a mechanical arm, or combined with a robotic operating mechanism.
[0037] The optical filter can be placed anywhere in the optical path between the source of the propagating light and the light-receiving end, such as a camera sensor.
[0038] Hyperspectral images may be acquired by an imaging device, which acquires sparse hyperspectral information by assigning spectral information of strictly fewer spectral bands than the total number of spectral bands that the imaging system can measure to each spatial position. An example of such a prior art imaging system is shown in U.S. Patent No. 9,857,222(B), which describes the use of a mosaic of filters for passing different bands of the optical spectrum and a sensor array arranged to detect pixels of the image in different bands that have passed through the filters. For each pixel, the sensor array has a cluster of sensor elements for detecting different bands, and the mosaic has a corresponding cluster consisting of different band filters, which are integrated on the sensor elements so that the image can be detected in different bands simultaneously.
[0039] An exemplary imaging system may include a sensor array of individual 5x5 mosaic sensors, as shown in Figure 1. The active sensor area is obtained by configuring an array of such individual mosaic sensors (see Figure 2). Each 5x5 mosaic sensor incorporates 25 optical filters sensitive to different spectral bands. This allows for sparse sampling of hyperspectral information across the active sensor area, with each spectral band information acquired only once per 5x5 area and spatially shifted relative to other bands. Images acquired by such a sensor array arrangement are called “mosaic” images or “snapshot mosaic” images.
[0040] In practice, due to the imperfections and physical design constraints of hyperspectral camera sensors, parasitic effects can lead to multimodal sensitivity response curves of optical filters. Examples of such effects affecting imaging include interference of higher-order spectral harmonics, out-of-band leakage, and crosstalk between adjacent pixels of the camera sensor. An exemplary response curve for a near-infrared (NIR) 5x5 mosaic sensor is shown in Figure 3. As will be evident, additional filters can be used to suppress or enhance the spectral portion of these responses. Reconstructing a simple hypercube from a snapshot mosaic image obtained by stacking images of bandwidth-related pixels results in a spatially and spectrally distorted hypercube representation with low resolution in both spatial and spectral dimensions. Therefore, snapshot hyperspectral imaging features high temporal resolution of hyperspectral images, while being affected by multimodal spectral bandwidth contamination and having low resolution in both spatial and spectral dimensions.
[0041] This specification discloses a method suitable for obtaining hyperspectral imaging information with high temporal and spatial resolution from snapshot imaging that provides wide-field and high-resolution tissue-related information in real time during surgery.
[0042] White balance adjustment and spatial spectrum calibration of acquired hyperspectral snapshot mosaic images may be performed as a preprocessing step using factory-acquired or user-acquired data. In some examples, this may be achieved by using a single image of a stationary object, such as a reflector, or a series of images of a stationary or moving object, acquired inside or outside the operating room. In some other examples, this may be done by processing specular reflections observed in the acquired image data. Further examples of image calibration may relate to image processing through intentional changes in imaging device settings, such as altering the effect of filter adjustments.
[0043] The process of reconstructing high-resolution hyperspectral image data from original low-resolution snapshot mosaics using image processing methods is called "demosaicing" or "upsampling." Demosaiking may be performed by spatially and spectrally upsampling the acquired snapshot mosaic data to obtain a hypercube, which has a fixed or optionally arbitrary number of spectral band information for all acquired image pixel locations. In some embodiments, such demosaiking may be performed to achieve spatial spectral upsampling onto a high-resolution grid other than the original image pixel locations. For example, in addition to the conventional demosaiking approach of performing spatial upsampling for resampling, we present spatial spectral-aware upsampling / demosaicing techniques that address both spatial crosstalk and spectral parasitic effects, which are particularly important in snapshot imaging systems. Such reconstructions are called "virtual hypercubes." A simple example of demosaiking is image resampling performed independently for each spectral band. Other examples may include methods based on inverse problem formulation. Other examples may include the use of data-driven, supervised, semi-supervised, or unsupervised / self-supervised machine learning approaches. These examples may include computational methods for reconstruction designed for irregular grids. Improvements in quality or robustness during mosaic removal can be achieved by processing video streams of image data. Similar approaches may be used to increase the temporal resolution of data visualizations.
[0044] Computational models may be used for parameter estimation from virtual hypercube representations. Examples include estimations from reflectance information of hyperspectral imaging, such as pixel-level tissue absorption information, which may include estimation of tissue properties at spatial location (typically at the pixel level). More generally, the obtained pixel-level hyperspectral information can be analyzed for the composition of intrinsic end-members characterized by specific spectral features. To derive tissue properties relevant to surgical guidance, spectral unmixing algorithms are presented to estimate the relative abundance of end-members mixed into the pixel spectrum. Examples of end-members include oxyhemoglobin and deoxyhemoglobin (Figure 4). Examples of tissue properties derived from relevant end-members include pixel-level hemoperfusion and oxygenation level saturation level information.
[0045] Other examples of unmixing may include estimation of fluorescence and autofluorescence, which may also be used for quantitative fluorescence.
[0046] In another embodiment, a virtual hypercube representation can help visualize hyperspectral image data by estimating a pseudo-red-green-blue (RGB) image or any other image of reduced dimensions.
[0047] In another embodiment, a virtual hypercube can also be used to classify pixels according to tissue types, including benign and malignant types. The virtual hypercube can also be used for semantic segmentation beyond tissue types, which may include the classification of any pixels related to non-human tissues, such as surgical instruments. The resulting segmentation can be used to improve the robustness of tissue parameter estimation or to correct for possible image artifacts, such as specular reflections.
[0048] In all the examples described here, the estimation and parameter extraction of the virtual hypercube can be performed in two independent steps or together. The computational model may use an algorithm that enables the joint implementation of mosaic removal and parameter estimation. Such an approach may be based on the formulation of an inverse problem, or on supervised or unsupervised machine learning approaches.
[0049] All computer-aided parameter estimations can be associated with estimations of uncertainty.
[0050] This disclosure relates to an image processing system and method that enables online video processing of high-resolution hypercube data from a video stream of sparse, low-resolution mosaic data acquired in real time by a medical device suitable for use in a sterile environment.
[0051] <System Description> Figure 5 shows an overview of an exemplary sterile imaging system that may be used for real-time hyperspectral imaging. A real-time hyperspectral imaging camera, such as a snapshot hyperspectral camera, is mounted on a sterile optics scope, such as a sterile exoscopy, via a suitable adapter. This adapter may also allow for zooming and focusing of the optical system and may include additional functions such as a mechanical shutter, beam splitter, or filter adapter. In some embodiments, the use of several camera sensors in combination with an optical splitting mechanism may be advantageous for covering the wavelength range of interest. For ease of explanation, such configurations may also be referred to as hyperspectral imaging cameras. The sterile optics scope is connected to a light source, such as a broadband xenon or LED light source. The light provided by this light source is of a spectral wavelength suitable for the hyperspectral imaging camera, or light that excites a fluorescent substance of interest via an optical guide that can be sterilized or draped. In some embodiments, the light source may be mounted together with the camera, which obviously may eliminate the need for an optical guide. Optical filters can be placed anywhere in the optical path between the source of the propagating light and the receiving end, such as a camera sensor. Such optical filters may be inserted using various means, such as a filter wheel in a light source capable of holding multiple optical filters, or they may be embedded in an adapter or endoscope. In some embodiments, optical filters may be used to eliminate unwanted out-of-band responses, such as a portion of the visible light from the NIR sensor (Figure 3). The hyperspectral imaging camera is connected to a computing workstation via a data link, such as a cable or wireless communication. Advantageously, power may be supplied to the camera sensor and other power-operated elements attached to the camera (e.g., tunable lenses or filters) via the same cable as the data link, similar to the case of a Power over Ethernet (PoE) connection. The workstation processes the acquired hyperspectral imaging information and can display the resulting information to the user via a display monitor.In some embodiments, the workstation may be embedded in a camera sensor unit or a display monitor. The visualized information may include acquired hyperspectral image data, or information obtained from that data by the calculation methods described below, such as RGB images or tissue characteristic information. By overlaying different types of information, more context can be presented to the user. For example, tissue characteristic information of a region estimated with high confidence can be overlaid on a pseudo-RGB rendering of the acquired (captured) scene. The sterility of the imaging system may be ensured by a combination of draping or sterilization of system components and procedural steps to ensure that connections between sterilized and non-sterile components do not impair the sterility of the sterilization operator and the field. One advantageous embodiment may involve using a sterile drape for a camera and data cable sealed in a sterile optic scope connected to a sterile light guide. The sterilized imaging system may be held by the user or fixed to an operating table, which allows for controlled movement or fixation of the imaging system at the user's request during surgery. The controlled mobility and immobilization of the sterilized imaging system can be achieved using a sterilized or draped mechanical arm or robotic actuation mechanism. In other embodiments, the hyperspectral imaging system may be embedded in a surgical microscope.
[0052] More generally, we present the key design requirements for HSIs for intraoperative surgical guidance suitable for open surgery. Following these criteria, we introduce and explain in more detail the iHSI invention embodied and illustrated in Figure 5.
[0053] <Design Requirements for Intraoperative HSI Systems> The initial design assumption is that the intraoperative application of the HSI camera system will be facilitated by developing a standalone, lightweight device that is independent of or complements the surgical microscope typically used in neurosurgery. In particular, ensuring compatibility with surgical telescopes and surgical microscopes such as exoscopies
[30] and endoscopes will enable a modular and flexible system design suitable for both open (laparotomy) and endoscopic surgery across surgical specialties. In accordance with this assumption, Tables 1 and 2 outline the design requirements considered to embody a hyperspectral imaging system for intraoperative surgical guidance, including minimum and target requirements. These are divided into (i) functional requirements, i.e., requirements arising from the clinical environment of the OR during surgery (Table 1), and (ii) technical requirements, i.e., specifications for the HSI system to achieve high-fidelity imaging data in order to meet the list of functional requirements for real-time surgical guidance (Table 2). Where objective requirements are not readily available, the best estimates based on the inventors' experience are given, as outlined below.
[0054] As part of the surgical requirements, it is beneficial that the iHSI system be sterilizable to allow for safe handling by the surgical team (F1), that it conforms to standard technical safety specifications (F2), that the lighting requirements do not interfere with the surgical workflow (F3), and that the device is easy to maintain and clean in accordance with standard surgical procedures (F4). Handheld devices are advantageous to be easy to operate, although it is advantageous that they be securely attached during operation, so that a single operator can control, move, and fix the imaging system without the need for an assistant (F5). The spatial resolution and spectral information acquired in the surgical images should, advantageously, be compatible with the surgical procedure, i.e., compatible with providing wide-field information that shows a minimum area and provides sufficient context for surgical decision-making (F6). In addition, it should, advantageously, facilitate the ability to monitor a wider range of tissues related to the surgery. The device should be advantageously able to provide important functional or semantic tissue information, and also be able to provide detailed information on multiple functions for comprehensive patient monitoring to enhance surgical precision and patient safety during surgery (F7). In the case of neurotumor surgery, this may be the tissue boundary to clearly show the relationship between tumor tissue and important brain structures such as nerves, blood vessels, or normal brain tissue. Furthermore, the image resolution should be advantageously detailed enough to facilitate spatial identification between tissue types within the surgical field (F8). The imaging should be advantageously displayed at video rate to facilitate immediate feedback to the surgeon and facilitate the seamless integration of workflows at higher video rates to enable a smoother experience (F9). Accurate visualization of the extracted information is advantageous for surgical guidance and can lead to a better user experience using an intuitive display system (F10).
[0055] To ensure surgical safety and sterility, it is advantageous and necessary that the system be easy to maintain and that its components can be effectively cleaned using standard antimicrobial surface wipes (Tl). The minimum advantageous requirements for the dimensions and weight of the HSI camera are based on estimates obtained by the prototyping-testing design thinking methodology
[31]
[29] , namely 10 × 10 × 12 cm 3 It is a smaller (T2) and lighter (T3) camera than the previous model. Its dimensions are 6 x 6 x 8 cm. 3For smaller systems, standard drapes can be used to cover the camera and ensure sterilization. Furthermore, all camera edges should, advantageously, be smooth to prevent tearing of the sterilization drape and injury to staff members (T4). A maximum camera temperature of 40°C advantageously ensures technical safety regarding the handling of the device, in addition to reducing dark current to maintain appropriate signal-to-noise ratios (SNR) during image acquisition (T5). The number of cables for power supply to the camera and data connection to the camera should, advantageously, be kept to a minimum (T6). To enable proper iHSI, a suitable light source should, advantageously, be available to provide sufficient energy across the active spectral range of the HSI camera (T7), but technical safety and photosafety considerations should be taken to ensure that light exposure does not harm the patient (T8). This is advantageously related to adhering to maximal permissible exposure (MPE), particularly in ionizing ultraviolet (UV) wavelengths below 400 nm
[32] . To enable optimal light source conditions for acquiring HSI information during surgery, the light source settings should be adjustable (F3 and T10). This may, advantageously, include adjusting optical filters to acquire high-fidelity HSI signal measurements, in addition to optimal light intensity settings according to the surgical scene, depending on the imaging requirements of the HSI camera. These settings can be adjusted by automatically addressing dynamic changes such as illumination in the OR. A stationary mount system is a minimal advantageous requirement to enable proper handling of the device during surgery (T9). Camera settings are advantageously adjustable according to the surgical context, enabling the acquisition of high-fidelity HSI information (T10). This involves implementing an adjustable system mount (T9), which enables automatic adjustment of camera settings (T10). By meeting favorable target requirements (T2, T3) for camera dimensions and weight, the handling of the device can be further improved.
[0056] High-fidelity tissue information can be effective if each target tissue is within the imaging field and in focus during HSI acquisition. This may require refocusing during surgery, either manually or using autofocus placement (Tll). A fixed field of view (FOV) of 40mm–60mm (T13) and a depth of field (DOF) of at least 20mm (T14) with a constant working distance (WD) between 200mm and 300mm (T12) is the minimum favorable requirement for iHSI
[33] , but a more favorable scenario includes systems with variable WD, variable FOV, and variable DOF to maximize compatibility with current surgical visualization systems
[34] . The number of spectral bands, spectral range, and spatial image resolution depend heavily on the clinical application to provide important functional and / or semantic features. Based on a review of previous literature and the inventors' own experience, providing several tens of clearly defined spectral bands is advantageous in achieving significant improvements over standard RGB imaging. Based on the availability of state-of-the-art industrial snapshot HSI sensors (see Table 3), an exemplary embodiment that achieves favorable requirements for an iHSI system during surgery may provide 16 spectral bands (T15) and a spectral range of at least 160 nm (T16). Other exemplary embodiments may, at a lower frame rate, advantageously achieve superior tissue identification capabilities by utilizing at least 100 spectral bands with a spectral range of at least 500 nm (see Table 3). Exemplary embodiments achieve at least 3 pixels per millimeter, with the favorable goal of providing information with an accuracy of at least 1 mm. Exemplary embodiments that achieve the requirements of a favorable minimum FOV and a favorable target FOV incorporate imaging grids of at least 120 × 120 and 450 × 450. However, based on currently available HSI sensor technology, substantially higher resolutions are possible.Therefore, exemplary embodiments are advantageous in targeting high resolution (1920 × 1080 pixels) and ultra-high resolution (3840 × 2160 pixels), respectively, due to advantageous minimum and target requirements (T17). Image calibration advantageously supports the generation of analyzable HSI data. Typical embodiments include acquiring both white and dark reference images for white balance to address ambient light and specific camera settings (T18). Examples of means typically include acquiring images using white reflective tiles and acquiring images with the shutter closed, respectively. However, for surgical guidance in OR, calibration data should be advantageously utilized so as not to interrupt the clinical workflow.
[0057] The minimum advantageous acquisition rate is beneficial because it is fast enough to provide real-time information suitable for surgical decision-making without interfering with the surgical workflow (T19). Based on the processing speed of the human visual system, image visualization rates faster than 7 frames per second (FPS) are advantageous
[35] . In some scenarios with static scenes, an image acquisition rate of a few seconds per image per surgical scene may be sufficient to provide the surgical team with important information. However, iHSI suitable for real-time image-guided surgery needs to be able to advantageously provide video-rate acquisition to provide live display of tissue information, which also allows for dynamic scenes during surgery.
[0058] <Intraoperative HSI System Design> In accordance with the above system design requirements, the inventors propose an iHSI system embodied and shown in Figure 5. The HSI camera can be advantageously connected to a sterilization optical scope via a suitable eyepiece adapter. The sterilization optical scope can be advantageously connected to a light source via a sterilization light guide.
[0059] The following are some specific aspects, features, and advantages of the embodiments relating to this disclosure. 1. To enhance the surgeon's vision and support decision-making during surgery, a surgical exoscopy is used in combination with an HSI camera for real-time tissue characterization. In some embodiments, the sterile optical scope may be an exoscopy, an endoscope, or a surgical microscope. 2. An optical filter is embedded in the HSI camera head to effectively filter the signal near the HSI camera sensor, enabling high-fidelity data acquisition. In some embodiments, an optical filter can be advantageously added to the eyepiece adapter to acquire high-fidelity hyperspectral images while enabling a compact optical system. 3. Tuned optical filter arrangements are used to account for differences in the quantum efficiency of individual band sensors and to obtain a uniform sensor response across all band sensors. In some examples, this may involve using a tuned optical filter arrangement to account for differences in the quantum efficiency of individual band sensors and to obtain a uniform sensor response across all band sensors for signal processing. 4. Embedding of filters to enable on-the-fly image calibration during surgery. Intentional alteration of the optical path can be achieved by adjusting the on-the-fly image calibration optical filter arrangement during surgery, provided the target has known reflectivity characteristics. In some embodiments, these filter adjustments can be performed using a filter wheel of the light source. 5. An infinity-corrected module is used to maintain the same beam focus (convergence) in the HSI camera sensor plane where multiple filters are arranged. In some examples, the infinity-corrected module may be used to generate parallel light beams between the objective lens and the tube lens. Since parallel-plane optical elements (e.g., filters) can be added without changing the cofocus, no image shift occurs. 6. Custom attachments for telescopes that interlock components of an optical system to prevent sliding, rotating, or any other modification, in order to prevent undesirable changes in the optical properties of the optical system during surgery. Components of an optical system, including filters, telescopes, and optical lenses, may be mechanically locked to prevent undesirable changes in the optical properties of the optical system during surgery. In some embodiments, this involves the use of custom attachments, such as connectors of a triangular or other geometric shape, which can mechanically lock the optical components and prevent sliding, rotating, or any other modification, thereby maintaining the desired optical properties for image calibration and processing. 7. A custom optical splitter splits light into spectrally separated beams to acquire multiple subimages on one or more HSI sensors. A custom optical splitter may be used to split light into spectrally separated beams to acquire multiple subimages on one or more HSI sensors. In some embodiments, beam splitting may be achieved using a microlens array or a prism such as a Wollaston prism, so that a single beam is split into two or more subimages on the same sensor. Advantageously, different optical filters can be used on different beams to acquire hyperspectral images. The use of hyperspectral sensors with multiple modes of spectral response can be advantageously combined with such a large number of filters, and regardless of the use of one or more sensors, responses targeting different subsets of spectral modes can be generated. 8. High-frequency light source switching for detecting and identifying the contribution of ambient light from acquired HSI data. Controlled on / off switching of a high-frequency light source can be used to detect and identify the contribution of ambient light from acquired HSI data. Synchronization between the light source switching and HSI camera acquisition can be achieved using another trigger mechanism, such as a separate cable connection. 9. Thermal dissipation measures suitable for surgical setups to ensure low-noise imaging and safe operator handling. Additional thermal dissipation measures, including active and passive cooling mechanisms, can be applied to the draped camera system to ensure low-noise imaging and safe operator handling. In some embodiments, this may include providing a heat sink directly attached to the camera. In other embodiments, this may include ensuring thermal conductivity between the camera and a mechanical arm or stand used to help position the imaging system.
[0060] In addition to connecting the scope to the camera, the eyepiece adapter may provide control mechanisms for zoom and focus. Filters can be placed anywhere in the optical path between the source of the propagating light and the receiving end, such as a camera sensor. In some embodiments, optical filters can be placed in a filter wheel embedded in the light source to restrict the light source spectrum according to the camera sensor or clinical requirements.
[0061] The HSI camera may be connected to a computing workstation or equivalent device via a connection that provides both power and a high-speed data link suitable for real-time HSI data transfer. This workstation or equivalent device processes the acquired HSI data and visualizes the derived information in real time. A sterile surgical drape covering both the HSI camera and the data cable may be sealed by a sterile exoscopy to ensure the sterility of the entire imaging system.
[0062] Depending on the surgical application, the sterile imaging system may be handheld by the operator, or it may be fixed to the operating table using a standard mechanical arm that allows for controlled movement or fixation of the imaging system, depending on the clinical requirements during surgery. In some embodiments, the camera system is sufficiently lightweight that its controlled movement and fixation can be performed using a sterile or draped mechanical arm attached to a sterile optics scope. Such a mechanism allows the iHSI system to be positioned at a safe distance outside the surgical cavity while the eyepiece adapter is able to focus properly for HSI data acquisition. Other embodiments may include the use of a robotic positioning arm to hold and control the imaging device.
[0063] The calculation step is performed by a computing workstation 52 to extract tissue or object characteristic information at the pixel level from low-resolution snapshot hyperspectral image data acquired for display during surgery. The computing workstation 52 is shown in detail in Figure 11, and as can be seen from the figure, the computing workstation 52 may be a appropriately programmed general-purpose computer and includes a processor 1128 with memory 1130 and an input / output interface 1132, etc. The input / output interface 1132 can receive control input from peripheral devices such as a keyboard, foot switch, pointing device (computer mouse or trackpad, etc.). A further input port 1134 is connected to a hyperspectral imaging camera for acquiring hyperspectral imaging data, and an image data output port 1136 is connected to a display for displaying images generated using the hyperspectral imaging data as input according to this embodiment.
[0064] A computer-readable storage medium 1112 such as a hard disk or a solid-state drive is also provided, which stores appropriate control software and data for operating the embodiments of the present invention. In particular, the storage medium 1112 stores operating system software 1114 that provides overall control of the computing system 52, and also stores a spatial-spectral mosaic removal program 1116 and a parameter estimation program 1118. Furthermore, a parameter mapping program 1120 is also provided. As described below, the spatial-spectral mosaic removal program 1116 and the parameter estimation program 1118 operate in conjunction to provide the first embodiment, while the parameter mapping program operates to combine the functions of the spatial-spectral mosaic removal program 1116 and the parameter estimation program 1118 into a single program to provide the second embodiment. Input to both embodiments is in the form of a plurality of snapshot mosaic images 1126, and output is various functional or semantic data images 1122 as described below. Furthermore, as described below, an intermediate data structure in the form of a virtual hypercube 1124 generated during the operation of the first embodiment can also be stored in the computer-readable medium.
[0065] <Detailed Description of Embodiments Relating to HSI Method> Hyperspectral information of an object such as tissue is affected by the parasitic effect of the spatial spectrum in addition to spatial-spectral downsampling during snapshot imaging. This results in obtaining a hyperspectral image characterized by low spatial resolution and low spectral resolution. The disclosed method can perform calculation steps to address each of these effects individually or jointly to obtain pixel-by-pixel estimates of characteristic information of tissue or an object with high spatial resolution.
[0066] By using filter response mapping, information about objects acquired by individual band sensors of a snapshot imaging sensor may be described, specifically hyperspectral information, which represents the lower spectral resolution of the objects. Furthermore, band selection and crosstalk modeling techniques may be used to describe the acquired low-spectral snapshot mosaic images and low-spatial-resolution snapshot mosaic images.
[0067] In some embodiments (i.e., the first embodiment described above), the hyperspectral information of a virtual hypercube, characterized by low spectral but high spatial resolution, can then be reconstructed by employing a spatial spectral correction approach. A computational parameter estimation approach may be used to infer the properties of the tissue or object on a pixel-by-pixel basis. Exemplary computational methods are disclosed below.
[0068] In other embodiments (i.e., the second embodiment described above), characteristic information of an organization or object can be directly obtained from the acquired low-resolution snapshot data. An exemplary calculation method is disclosed below.
[0069] A direct parameter estimation approach, which should be obvious to those skilled in the art, can be considered as an inference of tissue property information from a virtual hypercube corresponding to an estimated tissue property map, where the reconstructed virtual hypercube itself is the tissue property information. Further details are provided with reference to Figure 6.
[0070] Figure 6 outlines the relevant steps that can be used to perform the calculation methods of both the first and second embodiments. In summary, in the first embodiment, snapshot mosaic data W is captured in 62 and has low spatial resolution and low spectral resolution. This then undergoes a demosaicing process to generate a virtual hypercube 64 containing virtual high spatial but low spectral resolution data. Next, a parameter estimation process can be performed to obtain desired high spatial resolution data 66 in a desired parameter space from the virtual hypercube, as will be further detailed below.
[0071] In contrast, in the second embodiment, the snapshot mosaic data W is captured again at 62 and has low spatial and spectral resolution. This undergoes both the mosaic removal process and the parameter estimation process, and strictly speaking, refrains from the complete generation of the virtual hypercube (conceptually, it may be thought that the necessary parts are still generated even if the calculation is actually performed more directly). This is done to directly obtain the desired high spatial resolution data in the desired parameter space, as will be detailed below.
[0072]
number
[0073]
number
[0074]
number
[0075]
number
[0076]
number
[0077]
Number
[0078]
Number
[0079] As will be apparent to those skilled in the art, all calculation methods may also be used for multiple camera systems that provide multiplexed video stream data. In addition, all the calculation approaches presented may lead to reconstruction on different high-resolution grids other than the active sensor area W. All calculation methods may also be based on any other positive loss function of the norm represented in this example (for example, smooth L1 or bisquare). Specific assumptions regarding the noise level for error estimation may be made, for example, the assumption that noise is independent but not identically distributed across all wavelengths.
[0080]
Number
[0081] In some examples, the integration times τ and τ in (2) w may be identical. In other cases, τ may w be reduced to avoid potential sensor saturation effects. As will be apparent to those skilled in the art, a white reference may also preferably refer to any means for obtaining a spectrally neutral reference. In some embodiments, to avoid any potential saturation effects when obtaining a white reference, the use of a grey card requires, for example, τ wThis can be combined with an intensity correction coefficient similar to the effect of (2). A white reference may be estimated according to (2) using a sterile imaging target with known reflectivity properties, such as medical equipment available in an open operating room. An example of such a sterile imaging target may be surgical gauze. Specular reflections of one or more acquired images obtained from various angles and positions of the surgical scene may also be used as a substitute for the white reference signal.
[0082] In cases where imaging setting characteristics are known in advance, white balance adjustments can be calculated beforehand, eliminating the need to obtain white and dark references during surgery. Both white and dark references may be factory-estimated for various camera settings used for on-the-fly white balance adjustments during surgical use of the imaging system.
[0083] White balance adjustment according to (2) may also be performed for fluorescence imaging applications. This may include white balance adjustment of a system combined with optical components specifically designed for fluorescence-based imaging, such as an exoscopy with an appropriate light source and an optical filter for indocyanine green (ICG) or 5-aminolevulinic acid (5-ALA)-induced protoporphyrin IX (PpIX).
[0084] All white balance adjustment approaches presented may involve transient processing of the video stream. Such approaches may be used to address measurement uncertainties or to capture spatially varying white balance using non-uniform reflectivity targets such as surgical gauze. An example may be the time averaging of the white and dark reference images used in (2).
[0085]
number
[0086]
number
[0087]
number
[0088]
number
[0089] It is clear that this kind of constrained optimization model can be reformatted by including an additional regularization term in (3) and relaxing the strict constraints.
[0090]
number
[0091] In addition to commercially available calibration targets, sterile imaging targets with known reflectivity characteristics, such as medical devices available in open operating rooms, can be used in (6) u ref It is possible to define this.
[0092] (3)~(6) B F The initialization of the measured system filter response operator A F meas This can be carried out based on the following. One advantageous embodiment is to perform such initialization using data acquired in the factory and then refine it using data acquired in the clinical setting.
[0093]
number
[0094] Calibration of the optical system during surgical setup using (6) can also be achieved via a video stream to address measurement uncertainties, or via transient processing of the video stream to allow the use of spatially varying, non-uniform reflectivity targets such as surgical gauze for white balance adjustment. One example may include time averaging of acquired snapshot images w of a target having known mean reflectivity characteristics.
[0095]
number
[0096] Of particular note is that the calibration calculation in (7) assumes that the spatial crosstalk operator T is the identity operator, and n λ The method described in Pichette et al., Proc. of SPIE, 2017, applies only when m1m2 is true and all m1m2 bands are acquired at the same spatial location (Figure 7). This does not apply to snapshot mosaic imaging when imaging a spatially changing scene. Furthermore, a mosaic removal procedure that addresses both spatial crosstalk and spectral parasitic effects is not presented in Pichette et al., Proc. of SPIE, 2017.
[0097]
number
[0098] As will be apparent to those skilled in the art, all calibration methods described herein can be performed for multiple camera configurations, including different acquisition settings such as different gains.
[0099]
number
[0100] A simple and computationally fast approach to mosaic removal may be to use image resampling, such as linear or cubic resampling, on a calibrated mosaic image, followed by the application of the spectral calibration matrix C of (7). In the absence of a model that takes into account spatial and spectral parasitic effects, hypercube reconstruction will suffer from other artifacts such as blurring and edge shifts in both spatial and spectral dimensions, thereby increasing the uncertainty of subsequent tissue characterization (Figure 9).
[0101]
number
[0102]
number
[0103]
number
[0104] To improve computational efficiency, all operators may be implemented as operators without matrices.
[0105] Linear modeling using Tikhonov regularization is 2 When used in combination with norms, a dedicated linear least squares method such as LSMR may be employed to solve (10) or (11). For total variation-based regularization, the alternating direction method of multipliers (ADMM) can be used. Depending on the operator model, data loss, and the type and combination of regularization terms, other numerical approaches such as principal-dual or forward-reverse partitioning algorithms may be used as alternatives.
[0106] To accelerate the computation time of spatial spectrum-aware mosaic removal for real-time intraoperative guidance during surgery, a machine learning approach can be used that achieves fast computation time in inference at the expense of slow computation time during the training phase. The implementation of a machine learning approach can be based on a fully convolutional neural network (CNN). In some examples, the CNN can be implemented using an architecture such as U-Net.
[0107]
number
[0108]
number
[0109] In other embodiments, semi-supervised machine learning approaches may be used when an exhaustive or sufficiently representative / realistic database of paired examples is unavailable. Typical implementations of such approaches may rely on formulations combining supervised and unsupervised losses from (13) and (14). Examples may also include augmenting the dataset by synthesizing high-fidelity data pairs from available data using adversarial training approaches such as the use of generative adversarial networks (GANs).
[0110] The implementation of these examples may involve the use of deep neural networks such as CNNs. For example, it can be based on a single-image super-resolution reconstruction network architecture based on a residual network structure, where the initial upscaling layer considers regularly but spatially shifted hypercube sampling, which is reflected in the mosaic image acquisition. Other approaches may use input layers suitable for irregularly sampled input data, such as layers based on Nadaraya-Watson kernel regression.
[0111] <Temporally synchronized mosaic removal> Instead of reconstructing a virtual hypercube from a single mosaic image all at once, a time-synchronized approach may be employed to enhance robustness.
[0112] Spatial spectrum-aware mosaic removal can be used for temporally synchronized virtual hypercube reconstruction between two or more consecutive frames, and this can be based on motion compensation between frames.
[0113]
number
[0114] Machine learning-based supervised or unsupervised approaches for temporally matched spatial spectrum-aware mosaic removal can be based on video super-resolution approaches. These can be based on super-resolution networks with separated or integrated motion compensation, such as optical flow estimation. Another example that can be built on recurrent neural networks (RNNs), such as long short-term memory (LSTM) networks, to process video streams of transient snapshot image data.
[0115] A similar approach can be used to improve the temporal resolution of visualizations of data derived from snapshot mosaic images. In some examples, this can be done with half the time step t+1 / 2, w t+1 / 2 This can be done by estimating the displacements p / 2 and q / 2 obtained during optical flow estimation as in (15), and by doubling the frame rate of the HSI data visualization.
[0116]
number
[0117]
number
[0118] In one example, the regularization of (17) may be omitted, and the relative abundance can be simply calculated using the usual equations.
[0119] Other options for measuring the discrepancy between Ea and x in (17), such as cosine distance, can be used. The specific assumptions regarding the noise level in (16) may be such as the assumption that the noise is independent but not identically distributed across wavelengths.
[0120]
number
[0121]
number
[0122] Other examples of spectral unmixing may include tissue identification based on known spectral reflectance characteristics of tissue types.
[0123] A virtual hypercube that holds NIR reflectance information can also be used for spectral unmixing during fluorescence imaging based on known absorption and emission spectra of fluorescent compounds such as PpIX and ICG. This can also be used for quantitative fluorescence imaging to estimate the concentration of fluorescent compounds.
[0124] Pseudo-RGB images may be obtained from a virtual hypercube, for example, by using a CIE RGB color matching function (Figure 10c). If the virtual hypercube does not exhibit a spectral band that covers the visible spectrum for RGB reconstruction, or only partially covers it, a colorization / color regression method may be employed. Particularly in the case of NIR imaging, this may include supervised or unsupervised methods for colorization to estimate pixel-by-pixel RGB information. One example may include the use of unpaired samples of surgical RGB images and a periodic adversarial network for virtual hypercube reconstruction. Other approaches may be based on the use of higher-order responses of the optical system in the visible range. In one embodiment using an NIR imaging sensor, higher-order responses that are generally considered to be undesirable spectral responses outside the effective range of the sensor that need to be removed can be specifically used to obtain spectral measurements outside the NIR region (Figure 3). By switching between optical filters using known filter response curves of the sensor across the entire optical spectrum, signals of RGB color information covering the NIR or visible range can be sequentially obtained for image reconstruction. In some examples, such switching can be advantageously implemented by using a filter wheel embedded in the light source.
[0125] Other examples of parameter estimation may include segmentation of tissues or surgical tools using data-driven supervised, semi-supervised, or unsupervised / self-supervised machine learning approaches.
[0126] In other examples, the optical properties of a tissue can be estimated using a virtual hypercube and its pixel-by-pixel reflectivity information. One example might include absorption coefficients, which can be estimated using parametric models similar to the inverse additive duplication (IAD) or inverse Monte Carlo methods. For example, based on absorption estimates obtained using IAD, parametric model regression, etc., supervised or semi-supervised machine learning approaches can be devised to estimate an absorption map from the virtual hypercube.
[0127] Segmentation and parameter estimation methods can be used together. One example may include automated tissue segmentation, which can be used to provide tissue-specific scattering before obtaining more accurate absorption coefficient estimates.
[0128] Other examples may include automated segmentation of tissues or surgical tools for more robust tissue parameter estimation. This may include addressing image artifacts such as specular reflections or rejecting signal contributions unrelated to the tissue.
[0129] Other image analysis methods can be used to derive information related to surgical decision-making from the virtual hypercube.
[0130]
number
[0131] Such a model is an "ideal" parameter mapping h p :U→P n This can be based on prior knowledge.
[0132]
number
[0133]
number
[0134] Similar to the examples (10) to (14) above, parameter mapping f is performed using inverse problem-based approaches, supervised approaches, semi-supervised and unsupervised approaches. p :W→P n It is also possible to estimate this.
[0135]
number
[0136]
number
[0137] In other examples, a data-driven machine learning approach may be used to obtain pseudo-RGB images from snapshot images. One example is the use of a periodic adversarial network for non-paired samples of surgical RGB images and snapshot mosaic images.
[0138] <Estimation of Uncertainty> The presented forward models are typically well-defined, but the problem of estimating the inverse model, for example, the pixel-by-pixel reconstruction of tissue characteristic parameters from low-resolution snapshot image acquisition, is a challenge. p :W→P n These issues are generally ambiguous. It is clear that for all the computational approaches presented, additional uncertainty quantification capabilities can be introduced to estimate the uncertainty of the obtained results. These may include approaches such as dropout sampling, probabilistic inference, ensembles of estimators, or test-time augmentations.
[0139] In other embodiments, invertible mapping may be used in the presented model as obtained by an invertible neural network, which not only learns forward mapping but also establishes the corresponding inverse process. Such an approach may be used to recover a complete posterior distribution that can capture the uncertainty in the obtained solution estimate.
[0140] The uncertainty estimate may be displayed to a user, or used as part of a computational pipeline or visualization strategy. According to one example, the uncertainty estimate for parameter estimates can be used to select and display only estimates that satisfy a given certainty criterion such as a threshold.
[0141] <Detailed Description of Embodiments Relating to iHSI Systems> First, as an exemplary embodiment of the system according to the present invention, one that integrates two state-of-the-art industrial HSI cameras as part of an iHSI system setup is presented. Next, two iHSI embodiments are evaluated and scored against the presented design requirements (Tables 1 and 2). This is followed by performing controlled checkerboard experiments to demonstrate that reliable reflectance measurements are obtained in these embodiments using both HSI cameras. Ex vivo experiments demonstrate the reflection properties of various tissue types. In this embodiment, a standard tripod system for photography advantageously enables versatile imaging configurations in controlled environments. Finally, a successful case study on the clinical feasibility of ethically approved inpatients is described. The case study demonstrates the ability of a real-time iHSI embodiment to be seamlessly integrated into a surgical workflow while respecting OR clinical requirements such as sterility.
[0142] <Embodiments of HSI Systems> As part of the realization of the proposed iHSI invention, two hyperspectral imaging cameras were investigated (Table 3): (i) an embodiment of line-scan HSI using an Imec snapscan VNIR camera, i.e., in the visible (VIS) to near-infrared (NIR) region; and (ii) an embodiment of snapshot HSI using a Photonfocus MV0-D2048x1088-C01-HS02-160-G2 camera.
[0143] The line scan embodiment captures hypercube images with a maximum spatial resolution of 3650 × 2048 pixels over 150+ spectral bands between 470 nm and 900 nm. The imaging speed for acquiring a complete hypercube ranges from 2 s to 40 s, depending on the acquisition parameters, illumination, and imaging target. The camera dimensions, excluding optics, are 10 × 7 × 6.5 cm. 3 The weight is 0.58 kg. The line scan technology features a high SNR across the entire spectral range. The integrated shutter automatically measures dark current, so only a white reference image for image calibration needs to be manually acquired.
[0144] The Photonfocus camera employs an Imec snapshot mosaic CMV2K-SM5x5-NIR sensor that acquires 25 spectral bands in a 5x5 mosaic across the spectral range of 665nm to 975nm. With a sensor resolution of 2048x1088 pixels, hyperspectral data is acquired with a spatial resolution of 409x217 pixels per spectral band. Video-rate capture of snapshot data is achieved at speeds up to 50 FPS depending on the acquisition parameters. The camera dimensions, excluding the optics, are 3x3x5.4cm. 3 The weight is 0.08 kg.
[0145] The passive prototype cooling system is manufactured with rounded edges and features two heatsinks mounted on the sides of the camera to maintain operating temperature, thus enabling low image noise during image acquisition (Figure 12a). This increased the overall dimensions by approximately 3 cm in each direction and the weight by approximately 0.2 kg.
[0146] To provide broadband light, an Asahi Spectra MAX-350 light source (300W xenon lamp) was used. Depending on the experiment, either a VIS module or a UV-NIR mirror module providing light in the 385-740nm or 250-1050nm range was available. When using the UV-NIR mirror module, an additional 400nm long-pass filter (Asahi Spectra XUL0400) was placed in front of the mirror module to suppress ultraviolet (UV) light and improve the photosafety profile. In the case of Photonfocus cameras, a 670nm long-pass filter (Asahi Spectra XVL0670) was placed in the filter wheel to avoid signal contamination due to out-of-band sensor response during image acquisition, which is caused by the sensor's sensitivity to light in the VIS spectrum. The light intensity of the Asahi light source can be adjusted in integer increments between 5% and 100%. This light source is connected to a Karl Storz 0°VITOM surgical exoscopy 20916025AA via a Karl Storz fiber optic cable 495NCS, enabling imaging at a safe distance of 25cm to 75cm. A custom adapter was used to connect an optical guide to the Asahi light source. The exoscopy is mounted to each HSI camera via an RVA Synergies C-mount 18-35mm ZOOM endoscope coupler, providing additional manual zoom and focus mechanisms.
[0147] In all experimental calibrations, a white reference image was acquired using tiles with 95% reflectivity. For the Photonfocus camera, a separate dark reference image was acquired by closing the lens with a cap.
[0148] <Verification of the iHSI embodiment against the design specifications> Both the line scan and snapshot camera-based iHSI embodiments were evaluated for their suitability of the intraoperative settings to the design requirements described in Table 2. A summary of the evaluation is shown in Table 3.
[0149] Starting with system requirements, sterilization of both camera settings can be ensured using a combination of drapes and sterilization components (T2). However, from the specifications of each camera, it is clear that using snapshot cameras would allow for a more compact iHSI system due to the smaller dimensions and weight of the cameras (T2, T3). The xenon light source provides sufficient energy across the entire VIS and NIR spectral range using a UV-NIR mirror module, as shown in Figure 13 (T7) (250-1050 nm).
[0150] Light safety is advantageously ensured by blocking UV light using a 400nm long-pass filter (T8). The light source can be remotely configured using a serial communication protocol, and the filter wheel position and light intensity can be adjusted using customized software (T10). Similarly, both the line scan camera system and the snapshot camera system come with an API interface that enables remote control and software integration.
[0151] Device handling is crucial to ensure the safe attachment and movement of the camera system during surgery without negatively impacting surgical workflow and sterility (T2, T3, T9, T10). Because snapshot camera-based systems are compact, this can be easily achieved using a mechanical arm structure (T9). However, for line-scan camera-based systems, the same approach cannot be used due to weight and form factor. Attaching and rotating the camera system in a rotational position with its weight supported solely by the endoscopic adapter and mechanical arm was not deemed safe.
[0152] Both camera settings rely on the same optical settings and adapters, enabling imaging at a safe distance to the surgical cavity between 250mm and 750mm (T12). When using a circular 50mm FOV with a fixed working distance of 250mm for both systems, the depth of field for both systems is 35mm (T12-14) based on the exoscopy manufacturer's specifications
[33] . Using an endoscope adapter, the focus and zoom can be manually adjusted to provide sharp images at a specific focal length (Tll). In terms of HSI data quality, both the spatial and spectral image resolution of the line scan camera are far superior to that of the snapshot camera (T15, T17). In particular, in addition to the smaller spectral band sampled by the snapshot camera, additional post-processing methods such as mosaic removal to address sparse spatial sampling are required to obtain HSI data information with sufficiently high spatial resolution for tissue analysis, exceeding 409 × 217 pixels per spectral band (T15, T17). Line scan systems cover a wide spectral range in both the VIS and NIR regions, enabling rich feature extraction, whereas snapshot cameras provide only NIR spectral information. Furthermore, line scan technology allows for high-fidelity HSI signal measurements with a high signal-to-noise ratio. In contrast, signals acquired using snapshot imaging are characterized by the inclusion of multimodal spectral bands and crosstalk signals inherent in mosaic imaging sensors, which must be addressed. As a result, line scan systems may be able to extract a wider range of relevant surgical features. However, the acquisition speed of line scan cameras, between 2s and 40s per image, can disrupt the surgical workflow by failing to provide the video rate information necessary for real-time surgical guidance (T19). Motion artifacts are particularly likely to occur when imaging targets that are not stationary. In contrast, the high frame rates of snapshot cameras, up to 50 FPS, enable real-time visualization that can easily capture moving imaging targets (T19).For line scan cameras, image calibration can be performed by acquiring a white reference image using only the integrated shutter. For snapshot cameras, both the dark reference image and the white reference image must be T18. For both camera settings, a robust calibration approach that can accommodate changes in illumination and imaging scene is crucial for estimating reliable HSI information for surgical guidance during surgery.
[0153] Overall, in these embodiments, the imaging quality of line scan cameras was superior to that provided by snapshot cameras. However, given their form factor, more elaborate mounting mechanisms would favorably ensure safe and sterile handling of the camera during surgery. Furthermore, their relatively low imaging rate prevents them from capturing HSI data without interrupting the surgical workflow, which is critical for providing real-time information for seamless surgical guidance. Nevertheless, their imaging characteristics allow for the assurance of high-quality HSI in controlled settings. In contrast, video-rate snapshot cameras enable compact and sterile iHSI systems that can be integrated into surgical workflows using standard clinical machine arm structures. For reliable tissue analysis, image processing methods can favorably address the reduction in spatial and spectral image resolution, in addition to the signal quality degradation characteristic of mosaic snapshot sensors.
[0154] <Checkerboard Study: Verification of iHSI System Embodiments> Both line scan cameras and snapshot cameras were tested in combination with the proposed intraoperative optical system embodiment, namely the endoscopic adapter and exoscopy, and HSI data was acquired in controlled experiments using a datacolor SpyderCHECKR checkerboard with 48 color patches. In the experiments, an Asahi light source was used in combination with a UV-NIR module, providing light from 400 to 1050 nm in combination with a 400 nm long-pass filter. The reference spectrum was acquired using an Ocean Optics Maya 2000 Pro 200-1100 nm spectrometer with an Ocean Optics QR600-7-VIS125BX reflective probe (Figure 4).
[0155] Images were acquired using a line scan camera with an exposure time of 10 ms and a gain of 1.2. Images were acquired using a snapshot camera with an exposure time of 15 ms and a gain of 2. Using proprietary software and the default image calibration files provided with the cameras, spectrally calibrated hypercube reflectance data for image analysis of both camera systems was provided. In particular, dedicated system-wide calibration was not performed during image calibration to take into account specific light source intensity spectra (Figure 13) and individual optical components of the iHSI system (optical filters, endoscope adapters, exoscopy, etc.).
[0156] Both line scan and snapshot cameras were placed 35 cm away from the checkerboard, and images of each patch were acquired individually. For each calibrated hypercube image, five circular regions with a radius of 10 pixels, distributed across the color patch, were manually segmented for spectral analysis (Figure 14c).
[0157] Figure 15 shows a comparison of reference data and spectral information acquired by the iHSI system using line scan and snapshot cameras. It can be seen that the estimated reflectances from both the line scan and snapshot iHSI systems closely follow the reference measurements of the spectrometer.
[0158] <In vitro research: experiments with carcass calves> In vitro experiments to characterize tissues were conducted in a controlled environment using fresh calf carcasses at Balgrist University Hospital in Zurich, Switzerland, using both line-scan and snapshot-camera iHSI embodiments. Calf carcasses were selected because their anatomical tissue closely resembles that of the human spine.
[36]
[0159] For histological analysis, various types of tissues were exposed, including tendons, muscles, bones, joint capsules, dura mater, and spinal cord. A standard tripod system was favorably used to mount the iHSI camera system to obtain optimal orientation and positioning for imaging of cadaver tissue samples (Figure 16). Both the line scan and snapshot cameras were securely mounted using custom adapter plates with 1 / 4-20UNC and 3 / 8-16UNC screw holes (Figure 12b). A Thorlabs DCC3260C RGB camera was added to the experiment to provide high-resolution 1936×1216 RGB imaging. This camera, without optics, measures 2.9×3.5×4.4 cm. 3 It weighs 0.04 kg. Its C-mount camera lens mount allows it to be used with the same endoscope adapter as part of the same iHSI setup. Furthermore, its housing has a 1 / 4-20UNC screw hole suitable for attaching a quick-release tripod plate.
[0160] Imaging of exposed tissue using three cameras followed the scheme summarized in Figure 17. By using separate reference points visible and differentiable across the VIS and NIR spectra, it was ensured that images acquired with different cameras could be retrospectively aligned. For ease of handling during the experiment, a set of six pinheads in red, black, blue, white, green, and yellow, tied together with nylon thread, was used. The first HSI camera (line scan or snapshot camera) was positioned, and after adjusting the zoom and focus to image the tissue sample, the reference points (fiducials) were placed on the tissue to ensure they were within the FOV. Next, the reference points were removed from the scene for image acquisition and carefully returned to their original positions to avoid anatomical changes, and then a second image was acquired with the same HSI camera. Without touching the scene, the HSI camera was swapped with an RGB camera using the tripod's quick-release mechanism, and an RGB image of the tissue sample with the reference points was acquired. Subsequently, without making any changes to the scene, the RGB camera was swapped with a second HSI camera on the tripod. Typically, the camera position, zoom, and focus had to be fine-tuned to ensure that the target tissue was in focus and that a reference point was present within the FOV before acquiring the image. After carefully removing the reference point, another image of the same scene was acquired without making any other changes to the settings. For spectral analysis, the neurosurgeon manually annotated the associated tissue types on the pseudo-RGB line scan images obtained by assigning the red, green, and blue channels to wavelengths of 660 nm, 570 nm, and 500 nm, respectively. Alignment between all images was achieved using affine point-based registration by manually annotating circular reference points
[37] . The manual segmentation in the line scan image space was then propagated into the snapshot image space for analysis using the acquired point-based affine registration.
[0161] In in vitro experiments, only a VIS mirror module was available as a light source, providing light in the 385nm to 740nm range. For NIR imaging using a snapshot camera, an additional 670nm long-pass optical filter in the light source's filter wheel was enabled. For all scenes, the snapshot camera was set to a gain of 3.01 and an exposure time of 20ms, thereby performing video imaging and acquiring multiple images for each individual static scene. This resulted in an average of 18 snapshot mosaic images per scene, the average of which was used for spectral analysis. For the line scan camera, a gain of 2 and an exposure time of 20ms were used. The light intensity of the snapshot, line scan, and high-resolution RGB cameras was set to 100%, 100%, and 50%, respectively. All camera imaging was performed with the room lights off and the window blinds lowered to reduce the influence of background light. To simplify the imaging workflow, reference data for image calibration was acquired once each for both the line scan and snapshot cameras at the beginning and end of the experiment. Thus, the same white balance information for each HSI camera was used to calibrate all images associated with different anatomical locations.
[0162] Figure 18 provides a comparison of estimated reflectance curves between 470 nm and 740 nm for eight different anatomical scenes, as referenced in Figure 16b, using both line-scan and snapshot-based iHSI systems. For snapshot cameras, only five of the 23 reconfigured bands were available to analyze measurements between 670 nm and 740 nm. In general, the relative distribution and qualitative behavior of reflectance values across tissue types in overlapping spectral bands between cameras are well coordinated.
[0163] <Case study on clinical feasibility for patients: Spinal fusion> According to an evaluation of the proposed iHSI embodiments, combining quantitative and qualitative assessments, of both line scan and snapshot cameras against design requirements essential for surgery, the snapshot-based embodiment would advantageously provide real-time HSI that can be seamlessly integrated into the surgical workflow. To confirm this hypothesis, the inventors conducted a case study of clinical feasibility during surgery as part of a spinal fusion procedure at Balgrist University Hospital in Zurich, Switzerland. This study was approved by the Swiss Federal Ethics Committee (BASEC Nr:req-2019-00939).
[0164] Figure 5 shows a schematic diagram of an iHSI embodiment used in surgery. In addition to the aforementioned system components, a standard Karl Storz mechanical arm was used to securely mount the iHSI camera system to the operating table using a multi-joint L-shaped stand (28272HC) via a clamp jaw (28272UGK). Secure mounting to the snapshot HSI camera via the VITOM exoscopy was achieved by a suitable rotating socket (28172HR) and clamp cylinder (28272CN). Overall sterilization of the system was ensured by high-pressure sterilization of the mechanical arm, exoscopy, and light guide before surgery, and by draping the camera and accompanying cables.
[0165] The primary objective of the intraoperative clinical feasibility case study was to verify the integration of the system into a standard surgical workflow. To focus on this objective, the inventors chose to mimic a current optical camera system during surgery, using white light between 385 nm and 740 nm without a 670 nm long-pass filter. For the snapshot camera, a light source providing 100% light intensity was selected, with a gain of 4 and an exposure time of 20 ms. A laptop running customized software for real-time interaction with the camera system and data visualization was placed on a cart at a safe distance outside the sterile environment. When connected to a monitor in the OR, the captured video-rate HSI data was displayed live (Figure 19a). In particular, this allowed for immediate feedback and interaction with the surgical team, enabling the acquisition of focused data in the surgical region of interest by adjusting the camera position and orientation in addition to the endoscopic adapter settings. Using this setup, in vivo imaging was performed at eight different stages during surgery, acquiring HSI data for various tissue types including skin, adipose tissue, scar tissue, fascia, muscle, bone, pedicle screws, and dura mater (Figure 19b). Imaging of each anatomical tissue took place between 6s and 44s, minimizing disruption to the surgical workflow. Following a successful surgery with a seamless transition to acquiring HSI data, a final record was performed at 3 minutes and 16 seconds to obtain imaging data covering the surgical cavity.
[0166] <Discussion and Conclusion> Previous studies have highlighted the potential of HSI for intraoperative tissue characterization as a non-contact, non-ionizing, non-invasive, and label-free imaging modality. Despite numerous research studies exploring the clinical potential of HSI for surgery, to our knowledge, no HSI system has been presented that can provide real-time information for intraoperative surgical guidance while adhering to stringent clinical requirements such as sterilization and seamless integration into surgical workflows.
[0167] Here, we present an embodiment of an HSI system for intraoperative surgical guidance suitable for open surgery. However, the invention allows for adaptation to endoscopic and microsurgical procedures. Based on either line scan or snapshot techniques, we embodied the invention with two state-of-the-art industrial HSI camera systems and evaluated their suitability for surgical use. Based on criteria established by the inventors, we present embodiments of intraoperative HSI and provide scoring for these requirements considering both HSI cameras. Controlled checkerboard experiments were conducted to demonstrate that reliable reflectance measurements can be obtained in the proposed embodiments using both HSI cameras. In vitro experiments were performed to investigate the reflectivity properties of various tissue types, including tendons, muscles, bones, joint capsules, dura mater, and spinal cord, where both iHSI camera setups were mounted on standard tripod systems, allowing for diverse imaging configurations in a controlled environment. In particular, it was demonstrated that the line scan camera is a suitable setup for providing high-resolution data in both spatial and spectral dimensions across the entire VIS and NIR spectrum for in vitro tissue analysis. The iHSI system enables seamless and safe transitions at various stages of spinal fusion surgery, acquiring video-rate HSI data of multiple tissue types, including skin, adipose tissue, fascia, muscle, bone, pedicle screws, and dura mater. A case study on successful clinical feasibility demonstrated that the proposed iHSI system can be seamlessly integrated into the surgical workflow and provide wide-field video-rate HSI images while respecting critical clinical requirements such as sterility. By developing a data-driven information processing pipeline, such video-rate HSI data can be utilized to provide real-time wide-field tissue characterization for surgical guidance during surgery.
[0168] The proposed in vitro setup can be used in combined experiments to acquire both high-resolution line scans and low-resolution snapshot HSI data. This can advantageously provide crucial information for developing real-time mosaic demosing and tissue differentiation methods for snapshot HSIs.
[0169] As part of an exemplary embodiment, any compact camera conforming to the camera dimensions and weight requirements outlined in Table 1 can be easily integrated into the proposed iHSI system configuration. It is also beneficial to enable HSI acquisition using various working distances, fields of view, depth of field and focal depth (Table 2), thereby enabling the device to be integrated with various commercially available exoscopy surgical systems
[34] and ensuring equivalence with current microscopy standards
[38] .
[0170] Our in vivo clinical feasibility case study demonstrated that embodiments of our invention integrate well into standard surgical workflows and can capture HSI data. Overall, members of the surgical and open operating room teams found the tested embodiments easy to use, although routine training is necessary to ensure smooth operation during surgery. The tested embodiments did not raise safety concerns for team members, and the system's size, weight, and portability were acceptable for maintaining a smooth surgical workflow. [Table 1] [Table 2] [Table 3]
[0171] Various further modifications to the above examples, whether additions, deletions, or substitutions, will be apparent to those skilled in the art in order to provide additional examples, all of which are intended to be included in the attached claims.
[0172] <List of References> [1] I.J.Gerard, M.Kersten-Oertel, K.Petrecca, D.Sirhan, J.A.Hall, and D.L.Collins,“Brain shift in neuronavigation of brain tumors: A review,” Medical Image Analysis,vol.35,pp. 403-420, jan 2017.[Online]. Available: http: / / dx.doi.org / 10.1016 / j.media.2016.08.007 https: / / linkinghub.elsevier.com / retrieve / pii / S1361841516301566 [2] P.A.Helm, R.Teichman,S.L. Hartmann, and D.Simon, “Spinal Navigation and Imaging: History, Trends, and Future,” IEEE Transactions on Medical Imaging, vol.34, no.8, pp. 1738-1746, aug 2015. [Online]. Available: http: / / ieeexplore.ieee.org / document / 7008544 / [3] G.Lu and B.Fei, “Medical hyperspectral imaging: a review,” Journal of Biomedical Optics, vol.19, no.1, p. 010901, jan 2014。 [Online]. Available: http: / / biomedicaloptics.spiedigitallibrary.org / article.aspx?doi=10.1117 / 1.JBO.19.1.010901 [4] M.Halicek, H.Fabelo, S.Ortega, G.M.Callico, and B.Fei, “In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques: Revealing the Invisible Features of Cancer,” Cancers, vol.11, no.6, p. 756, may 2019. [Online]. Available: https: / / www.mdpi.com / 2072-6694 / 11 / 6 / 756 [5] J.Shapey, Y.Xie, E.Nabavi, R.Bradford, S.R. Saeed, S.Ourselin, and T.Vercauteren, “Intraoperative multispectral and hyperspectral label-free imaging: A systematic review of in vivo clinical studies,” Journal of Biophotonics, vol.12, no.9, p.E201800455, sep 2019. [Online]. Available: https: / / onlinelibrary.wiley.com / doi / abs / 10.1002 / jbio.201800455 [6] N.T. Clancy, G.Jones, L.Maier-Hein, D.S. Elson, and D.Stoyanov, “Surgical spectral imaging,” Medical Image Analysis, vol.63, p. 101699, 2020. 〔Online〕. Available: https: / / doi.org / 10.1016 / j.media.2020.101699 [7] J.H. G.M. Klaessens, M.Nelisse, R.M. Verdaasdonk, and H.J. Noordmans, “Non-contact tissue perfusion and oxygenation imaging using A LED based multispectral and A thermal imaging system, first results of clinical intervention studies,” in Advanced Biomedical and Clinical Diagnostic Systems XI, A.Mahadevan-Jansen, T.Vo-Dinh, and W.S. Grundfest, Eds., vol. 8572, mar 2013, p. 857207. [Online]. Available: http: / / proceedings.spiedigitallibrary.org / proceeding.aspx?doi=10.1117 / 12.2003807 [8] M.Mori, T.Chiba, a. Nakamizo, R.Kumashiro, M.Murata, T.Akahoshi, M.Tomikawa, Y.Kikkawa, K.Yoshimoto, M.Mizoguchi, T.Sasaki, and M.Hashizume, “Intraoperative visualization of cerebral oxygenation using hyperspectral image data: A two-dimensional mapping method,” International Journal of Computer Assisted Radiology and Surgery, vol.9, no.6, pp. 1059-1072, nov 2014. [Online]. Available: http: / / link.springer.com / 10.1007 / S11548-014-0989-9 [9] e.Kho, LL deBoer, KK Van de Vijver, F.van Duijnhoven, M.-JT Vrancken Peeters, HJ Sterenborg, and TJ Ruers, “Hyperspectral Imaging for Resection Margin Assessment during Cancer Surgery,” Clinical Cancer Research, vol.25, no.12, pp. 3572-3580, Jun 2019. [Online]. Available: http: / / clincancerres.aacrjournals.org / lookup / doi / 10.1158 / 1078-0432.CCR-18-2089
[10] H.Fabelo, S.Ortega, R.Lazcano, D.Madronal, GM Callico, e.Juarez, R.Salvador, D.Bulters, H.Bulstrode, a.Szolna, J.Pineiro, c.Sosa, aJ O'Shanahan, S.Bisshopp, M.Hernandez, J.Morera, D.Ravi, B.Kiran, a.Vega, a.Baez-Quevedo, G.-Z. Yang, B.Stanciulescu, and R.Sarmiento, “An Intraoperative Visualization System Using Hyperspectral Imaging to Aid in Brain Tumor Delineation、” Sensors, vol.18, no.2, p. 430, Feb 2018. [Online]. Available: http: / / www.mdpi.com / 1424-8220 / 18 / 2 / 430
[11] N.Chiang, J.K. Jain, J.Sleigh, and T.Vasudevan, “Evaluation of hyperspectral imaging technology in patients with peripheral vascular disease,” Journal of Vascular Surgery, vol.66, no.4, pp. 1192-1201, oct 2017. [Online]. Available: http: / / dx.doi.org / 10.1016 / j.jvs.2017.02.047 https: / / linkinghub.elsevier.com / retrieve / pii / S0741521417309308
[12] M.Desjardins, J.-P. Sylvestre, R.Jafari, S.Kulasekara, K.Rose, R.Trussart, J.D. Arbour, c.Hudson, and F.Lesage, “Preliminary investigation of multispectral retinal tissue oximetry mapping using A hyperspectral retinal camera,” Experimental Eye Research, vol.146, pp. 330-340, may 2016. [Online]. Available: https: / / linkinghub.elsevier.com / retrieve / pii / S0014483516300653
[13] L.c. Cancio, a.I. Batchinsky, J.R. Mansfield, S.Panasyuk, K.Hetz, D.Martini, B.S. Jordan, B.Tracey, and J.e. Freeman, “Hyperspectral Imaging: a New Approach to the Diagnosis of Hemorrhagic Shock,” The Journal of Trauma: Injury, Infection, and Critical Care, vol.60, no.5, pp. 1087-1095, may 2006. [Online]. Available: http: / / journals.lww.com / 00005373-200605000-00025
[14] L.Khaodhiar, T.Dinh, K.T. Schomacker, S.V. Panasyuk, J.e. Freeman, R.Lew, T.Vo, a.a. Panasyuk, c.Lima, J.M. Giurini, T.e. Lyons, and a.Veves, “The Use of Medical Hyperspectral Technology to Evaluate Microcirculatory Changes in Diabetic Foot Ulcers and to Predict Clinical Outcomes,” Diabetes Care, vol.30, no.4, pp. 903-910, apr 2007. [Online]. Available: http: / / care.diabetesjournals.org / cgi / doi / 10.2337 / dc06-2209
[15] B.Fei, G.Lu, X.Wang, H.Zhang, J.V. Little, M.R. Patel, c.c. Griffith, M.W. El-Diery, and a.Y. chen, “Label-free reflectance hyperspectral imaging for Tumor margin assessment: A pilot study on surgical specimens of cancer patients,” Journal of Biomedical Optics, vol.22, no.08, p.1, aug 2017. [Online]. Available: https: / / www.spiedigitallibrary.org / journals / journal-of-biomedical-optics / volume-22 / issue-08 / 086009 / Label-free-reflectance-hyperspectral-imaging-for-tumor-margin-assessment / 10.1117 / 1.JBO.22.8.086009.full
[16] H.Fabelo, M.Halicek, S.Ortega, M.Shahedi, a.Szolna, J.Pineiro, c.Sosa, a.O’Shanahan, S.Bisshopp, c.Espino, M.Marquez, M.Hernandez, D.Carrera, J.Morera, G.Callico, R.Sarmiento, and B.Fei, “Deep Learning-Based Framework for In Vivo Identification of Glioblastoma Tumor using Hyperspectral Images of Human Brain,” Sensors, vol.19, no.4, p.920, feb 2019. [Online]. Available: http: / / www.mdpi.com / 1424-8220 / 19 / 4 / 920
[17] S.Gioux, a.Mazhar, B.T. Lee, S.J. Lin, a.M. Tobias, D.J. Cuccia, a.Stockdale, R.Oketokoun, Y.Ashitate, e.Kelly, M.Weinmann, N.J. Durr, L.a. Moffitt, a.J. Durkin, B.J. Tromberg, and J.V. Frangioni, “First-in-human pilot study of A spatial frequency domain oxygenation imaging system,” Journal of Biomedical Optics, vol.16, no.8, p. 086015, 2011. [Online]. Available: http: / / biomedicaloptics.spiedigitallibrary.org / article.aspx?doi=10.1117 / 1.3614566
[18] S.L. Best, a.Thapa, N.Jackson, e.Olweny, M.Holzer, S.Park, e.Wehner, K.Zuzak, and J.a. Cadeddu, “Renal Oxygenation Measurement During Partial Nephrectomy Using Hyperspectral Imaging May Predict Acute Postoperative Renal Function,” Journal of Endourology, vol.27, no.8, pp. 1037-1040, aug 2013. [Online]. Available: http: / / www.liebertpub.com / doi / 10.1089 / end.2012.0683
[19] H.J. Noordmans, c.Ferrier, R.deRoode, F.Leijten, P.van Rijen, P.Gosselaar, J.Klaessens, and R.Verdaasdonk, “Imaging the seizure during surgery with A hyperspectral camera,” Epilepsia, vol.54, no.11, pp. e150-e154, nov 2013. [Online]. Available: http: / / doi.wiley.com / 10.1111 / epi.12386
[20] M.Barberio, F.Longo, c.Fiorillo, B.Seeliger, P.Mascagni, V.Agnus, V.Lindner, B.Geny, a.-L.Charles, I.Gockel, M.Worreth, a.Saadi, J.Marescaux, and M.Diana, “HYPerspectral Enhanced Reality (HYPER): A physiology-based surgical guidance tool,” Surgical Endoscopy, vol.34, no.4, pp. 1736-1744, apr 2020. [Online]. Available: https: / / doi.org / 10.1007 / S00464-019-06959-9 http: / / link.springer.com / 10.1007 / S00464-019-06959-9
[21] J.Yoon, J.Joseph, D.J. Waterhouse, a.S. Luthman, G.S.D. Gordon, M.di Pietro, W.Januszewicz, R.c. Fitzgerald, and S.e. Bohndiek, “a clinically translatable hyperspectral endoscopy (HySE) system for imaging the gastrointestinal tract,” Nature Communications, vol.10, no.1, p. 1902, dec 2019. [Online]. Available: http: / / dx.doi.org / 10.1038 / S41467-019-09484-4 http: / / www.nature.com / articles / S41467-019-09484-4
[22] R.Muhle, H.Ernst, S.B. Sobottka, and U.Morgenstern, “Workflow and hardware for intraoperative hyperspectral data acquisition in neurosurgery,” Biomedical Engineering / Biomedizinische Technik, jul 2020. [Online]. Available: https: / / www.degruyter.com / view / journals / bmte / ahead-of-print / article-10.1515-bmt-2019-0333 / article-10.1515-bmt-2019-0333.xml
[23] a.Kulcke, a.Holmer, P.Wahl, F.Siemers, T.Wild, and G.Daeschlein, “a compact hyperspectral camera for measurement of perfusion parameters in medicine,” Biomedical Engineering / Biomedizinische Technik, vol.63, no.5, pp. 519-527, oct 2018. [Online]. Available: http: / / www.degruyter.com / view / j / bmte.2018.63.issue-5 / bmt-2017-0145 / bmt-2017-0145.xml
[24] H.Kohler, a.Kulcke, M.Maktabi, Y.Moulla, B.Jansen-Winkeln, M.Barberio, M.Diana, I.Gockel, T.Neumuth, and c.Chalopin, “Laparoscopic system for simultaneous high-resolution video and rapid hyperspectral imaging in the visible and near-infrared spectral range,” Journal of Biomedical Optics, vol.25, no.08, aug 2020. [Online]. Available: https: / / www.spiedigitallibrary.org / journals / journal-of-biomedical-optics / volume-25 / issue-08 / 086004 / laparoscopic-system-for-simultaneous-high-resolution-video-and-rapid-hyperspectral / 10.1117 / 1.JBO.25.8.086004.full
[25] J.Pichette, a.Laurence, L.Angulo, F.Lesage, a.Bouthillier, D.K. Nguyen, and F.Leblond, “Intraoperative video-rate hemodynamic response assessment in human cortex using snapshot hyperspectral optical imaging,” Neurophotonics, vol.3, no.4, p. 045003, oct 2016. [Online]. Available: http: / / neurophotonics.spiedigitallibrary.org / article.aspx?doi=10.1117 / 1.NPh.3.4.045003
[26] M.EwerloF, M.Larsson, and e.G. Salerud, “Spatial and temporal skin blood volume and saturation estimation using A multispectral snapshot imaging camera,” in Imaging, Manipulation, and Analysis of Biomolecules, Cells, and Tissues XV , D.L. Farkas, D.V. Nicolau, and R.c. Leif, Eds., vol. 10068, feb 2017, p. 1006814. [Online]. Available: http: / / proceedings.spiedigitallibrary.org / proceeding.aspx?doi=10.1117 / 12.2251928
[27] K.Dijkstra, J.van de Loosdrecht, L.R.B. Schomaker, and M.a. Wiering, “Hyperspectral demosaicking and crosstalk correction using deep learning,” Machine Vision and Applications, vol.30, no.1, pp. 1-21, feb 2019. [Online]. Available: https: / / doi.org / 10.1007 / S00138-018-0965-4 http: / / link.springer.com / 10.1007 / S00138-018-0965-4
[28] G.Tsagkatakis, M.Bloemen, B.Geelen, M.JayapalA, and P.Tsakalides, “Graph and Rank Regularized Matrix Recovery for Snapshot Spectral Image Demosaicing,” IEEE Transactions on Computational Imaging, vol.5, no.2, pp. 301-316, jun 2019. [Online]. Available: https: / / ieeexplore.ieee.org / document / 8584100 /
[29] J.Shapey, Y.Xie, e.Nabavi, D.Ravi, S.R. Saeed, R.Bradford, S.Ourselin, and T.Vercauteren, “Towards intraoperative hyperspectral imaging: design considerations for neurosurgical applications,” Proceedings of Hamlyn Symposium on Medical Robotics, p.18, 2018.
[30] L.Ricciardi, K.L. Chaichana, a.Cardia, V.Stifano, Z.Rossini, a.Olivi, and c.L. Sturiale, “The exoscope in Neurosurgery: An Innovative “Point of View”. a Systematic Review of the Technical, Surgical, and Educational Aspects,” World Neurosurgery, vol.124, pp. 136-144, apr 2019. [Online]. Available: https: / / doi.org / 10.1016 / j.wneu.2018.12.202 https: / / linkinghub.elsevier.com / retrieve / pii / S1878875019300804
[31] P.G. Yock, S.Zenios, J.Makower, T.J. Brinton, U.N. Kumar, F.T.J. Watkins, L.Denend, T.M. Krummel, and c.Q. Kurihara, Biodesign: The process of innovating medical technologies, 2nded. Cambridge University Press, 2015.
[32] S.H. Yun and S.J.J. Kwok, “Light in diagnosis, therapy and surgery,” Nature Biomedical Engineering, vol.1, no.1, p. 0008, jan 2017. [Online]. Available: http: / / www.nature.com / articles / S41551-016-0008
[33] K.Nishiyama, “From Exoscope into the Next Generation,” Journal of Korean Neurosurgical Society, vol.60, no.3, pp. 289-293, may 2017. [Online]. Available: http: / / jkns.or.kr / journal / view.php?doi=10.3340 / jknS.2017.0202.003
[34] D.J. Langer, T.G. White, M.Schulder, J.a. Boockvar, M.Labib, and M.T. Lawton, “Advances in Intraoperative Optics: a Brief Review of Current Exoscope Platforms,” Operative Neurosurgery, vol.19, no.1, pp. 84-93, jul 2020. [Online]. Available: https: / / academic.oup.com / ons / article / 19 / 1 / 84 / 5569799
[35] S.Thorpe, D.Fize, and c.Marlot, “Speed of processing in the human visual system,” Nature, vol.381, no.6582, pp. 520-522, jun 1996. [Online]. Available: http: / / www.nature.com / articles / 381520a0
[36] P.c. Cotterill, J.P. Kostuik, G.D’Angelo, G.R. Fernie, and B.e. Maki, “An anatomical comparison of the human and bovine thoracolumbar spine,” Journal of Orthopaedic Research, vol.4, no.3, pp. 298-303, 1986. [Online]. Available: http: / / doi.wiley.com / 10.1002 / jor.1100040306
[37] a.Myronenko and X.Song, “Point SET Registration: Coherent Point Drift,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.32, no.12, pp. 2262-2275, dec 2010. [Online]. Available: http: / / arxiv.org / abs / 0905.2635%0Ahttp: / / dx.doi.org / 10.1109 / TPAMI.2010.46 http: / / ieeexplore.ieee.org / document / 5432191 /
[38] J.Schwiegerling, c.Chancy, and R.c. Dimalanta, “Depth of Focus Measurement of an Ophthalmic Surgical Microscope,” Investigative Ophthalmology & Visual Science, vol.56, no.7, p. 1925, 2015.
[39] L.Ayala, S.Seidlitz, a.Vemuri, S.J. Wirkert, T.Kirchner, T.J. Adler, c.Engels, D.Teber, and L.Maier-Hein, “Light source calibration for multispectral imaging in surgery,” International Journal of Computer Assisted Radiology and Surgery, vol.15, no.7, pp. 1117-1125, jul 2020. [Online]. Available: https: / / doi.org / 10.1007 / S11548-020-02195-y http: / / link.springer.com / 10.1007 / S11548-020-02195-y
Claims
1. A method for determining the parameters of a desired target image from hyperspectral images acquired during surgical procedures, A process for capturing hyperspectral snapshot mosaic images of a surgical scene in real time using a hyperspectral image sensor connected to an optical scope, wherein the snapshot mosaic image comprises snapshot mosaic image data and has relatively low spatial and spectral resolution. A step of generating a virtual hypercube of the snapshot mosaic image data, wherein the mosaic removal is performed on the snapshot mosaic image data with regard to the spatial spectrum, the mosaic removal includes an algorithm that optimizes a cost function that takes into account spatial crosstalk between adjacent pixels corresponding to different spectral bands in the snapshot mosaic image, upsampling of the snapshot mosaic image data, and application of a spectral calibration operator, and the virtual hypercube comprises image data having a relatively higher spatial resolution than the snapshot mosaic image. A step of determining a relatively high spatial resolution parameter for a desired target image from the image data of the virtual hypercube, The process involves outputting the determined relatively high resolution parameters in real time as representing the desired target image, A method by which a computer performs a process that includes [something].
2. The aforementioned mosaic removal includes machine learning. The method according to claim 1.
3. The aforementioned mosaic removal is temporally consistent across two or more consecutive frames based on motion compensation between frames. The method according to claim 1.
4. The process further includes performing a white balance adjustment calculation on the hyperspectral image sensor before capturing the hyperspectral snapshot mosaic image. The method according to claim 1.
5. The aforementioned white balance adjustment calculation is performed over an integrated time τ ｄ and τ ｗ The process includes the steps of individually acquiring reference images, each containing a dark reference mosaic image and a white reference mosaic image, and executing a linear model, wherein the acquired mosaic image w of the object has an integration time τ. τ In addition, the integration time τ ｗ The white reference mosaic image and the dark reference mosaic image of the reference tile are integrated at time τ and τ ｗ The shutter is closed and the image is acquired, and the white balance adjustment calculation generates a reflective mosaic image given by the following formula. [Math A] The method according to claim 4.
6. The process further includes performing spatial spectral calibration calculations on the hyperspectral image sensor before capturing the aforementioned hyperspectral snapshot mosaic image. The method according to claim 1.
7. The real spectral filter response operator and spatial crosstalk operator T:W→W shown in the following equation are estimated in a controlled setup to address parasitic effects during image acquisition. [Number B] The method according to claim 6.
8. Snapshot mosaic image data is acquired using collimated light, and all n associated with a known imaging target that typically has spatially constant spectral features. Λ The process further includes a step of measuring the characteristics of the hyperspectral image sensor in order to obtain the measured system filter response operator shown by the following formula by sweeping the wavelength. [Number C] The method according to claim 7.
9. The step of determining the relatively high spatial parameters further comprises analyzing pixel-level hyperspectral information for the composition of intrinsic edge components characterized by specific spectral features. The method according to claim 1.
10. The process of determining the relatively high spatial parameters further comprises a step of estimating tissue characteristics for each spatial position (typically at the pixel level) from reflectance information of hyperspectral imaging, such as tissue absorption information at the pixel level. The method according to claim 1.
11. A method for determining the parameters of a desired target image from a hyperspectral image, A process for capturing a hyperspectral snapshot mosaic image of a scene using a hyperspectral image sensor, wherein the snapshot mosaic image has relatively low spatial and spectral resolution. A process for determining a relatively high spatial resolution parameter of a desired target image, comprising: a process for performing both mosaic removal and parameter estimation from the snapshot mosaic image, wherein the mosaic removal is spatial spectrum-aware and includes an algorithm for optimizing a cost function that takes into account spatial crosstalk between adjacent pixels corresponding to different spectral bands in the snapshot mosaic image; and a process for upsampling the snapshot mosaic image and applying a spectral calibration operator. A step of outputting the determined relatively high resolution parameter as representing the desired target image, A method by which a computer performs a process that includes [something].
12. A system for determining the parameters of a desired target image from a hyperspectral image, A hyperspectral image sensor is positioned to capture a hyperspectral image of the scene, Processor and A computer-readable storage medium for storing computer-readable instructions, wherein, when executed by the processor, the computer-readable instructions cause the processor to control the system and execute the method described in any one of claims 1 to 11. system.
13. The computer program, when executed, causes the hyperspectral imaging system to perform the method according to any one of claims 1 to 11, and stores such a program. Computer-readable storage medium.
Citation Information
Patent Citations
Endoscope apparatus
JP1996280610A
Light source device for medical apparatus and endoscope apparatus
JP2012245349A
Integrated circuits for spectral imaging systems
JP2013512445A
Compact light sensors for surgical applications and shock detection
US20190069824A1
Hyperspectral 2d imaging device
WO2017086788A1