Assisting a medical procedure with a luminescence image processed in a limited information region identified in a corresponding auxiliary image

Abstract

A solution for assisting a medical procedure is presented. A corresponding method comprises acquiring a fluorescence image (205F) of a field of view (103) based on fluorescence light and an auxiliary image (205R) of the field of view (103) based on auxiliary light different from the fluorescence light; the field of view (103) contains a region of interest comprising a target object (containing a fluorescence substance) and one or more foreign objects of the medical procedure. An auxiliary information region (210Ri) representing the region of interest without the foreign objects is identified in the auxiliary image (205R) according to its content, and a fluorescence information region (210Fi) is identified in the fluorescence image (205F) according to the auxiliary information region (210Ri). The fluorescence image (205F) is processed restricted to the fluorescence information region (210Fi) for facilitating identification of a representation of the target object therein. A computer program and a corresponding computer program product for implementing the method are also presented. Furthermore, a computing device for performing the method and an imaging system comprising it are presented. Further, a medical procedure based on the same solution is presented.

Description

Technical Field

[0001] This disclosure relates to imaging applications. More specifically, this disclosure relates to cold light imaging for assisting medical procedures. Background Technology

[0002] In the following text, the background of this disclosure is introduced by discussing techniques relevant to the context of this disclosure. However, even when the discussion involves documents, actions, artifacts, etc., it does not imply or represent that the techniques discussed are part of the prior art or common general knowledge in the field related to this disclosure.

[0003] Cold light imaging, particularly fluorescence imaging, is a specific imaging technique used to acquire images that provide a visual representation of an object, even if the image is not directly visible. Cold light imaging is based on cold light phenomena, including the emission of light by cold light substances when subjected to any excitation other than heating; in particular, fluorescence occurs in fluorescent substances (called fluorophores), which emit (fluorescent) light when illuminated.

[0004] Imaging techniques are commonly used in medical devices to examine a patient's (internal) body parts, thereby aiding in medical procedures. For example, in fluorescence-guided surgery (FGS) involving tumors (also known as fluorescence-guided resection (FGR)), a fluorescent agent (possibly a specific molecule adapted to reach a desired target such as a tumor and then held in place) is applied to the patient; the visualization of the fluorescent agent in the corresponding fluorescence image (usually superimposed with a corresponding reflectance image) facilitates the surgeon's manipulation, such as identifying the tumor to be removed.

[0005] However, the imagined field of view often includes several foreign objects beyond the actual area of ​​interest. For example, in (fluorescence)-guided surgery, this could be due to the presence of surgical instruments, hands, surgical tools, surrounding body parts (e.g., skin around the surgical cavity or unrelated organs within the surgical cavity), and background material, in addition to the fluorescence of actual interest emitted by the fluorescent agent accumulated in the target of the medical procedure (the tumor in the example discussed). Foreign objects can also generate false light. Specifically, false light may increase fluorescence. This could be due to scattering and absorption phenomena, for example; furthermore, it could be due to the accumulation of the fluorescent agent in surrounding body parts (particularly the skin), and due to (undesired) affinity for them. Conversely, false light may aberrantly or artificially reduce fluorescence.

[0006] False light (besides being ineffective) is detrimental to imaging tumors (or any other target). Specifically, false light significantly biases the statistical distribution of fluorescence values ​​in a fluorescence image; this adversely affects subsequent processing of the fluorescence image. For example, fluorescence values ​​are typically converted from a range given by fluorescence measurements to a range given by the display dynamics of the monitor used to display the fluorescence image. Therefore, the bias of the statistical distribution of fluorescence values ​​by false light (when fluorescence is increased) limits the range of fluorescence values ​​used to display the region of interest. This makes the representation of tumors within it less significant. Furthermore, fluorescence images are often thresholded based on a comparison of their fluorescence values ​​to a threshold to distinguish tumors from other parts of the fluorescence image; the threshold is calculated automatically based on the fluorescence values. In this case, the bias of the statistical distribution of fluorescence values ​​by false light (increasing or decreasing fluorescence) also affects the threshold. This involves the risk of misclassifying fluorescence values. As a result, there may be overdetection of tumors (false positives), and especially underdetection of tumors (false negatives); furthermore, this hinders the detection of tumor lesions. All of the above situations can have serious consequences for the patient's health (such as incomplete tumor removal).

[0007] The contribution of false light is difficult (if not impossible) to remove from the fluorescence of interest. In fact, statistical techniques are largely ineffective for this purpose because false light is not statistically distinguishable. Optical filtering is also largely ineffective, especially when false light has the same spectral characteristics as the fluorescence of interest (such as in cases where fluorescent agents have accumulated in the skin). Furthermore, manually adjusting the operating parameters of the medical device to limit the effects of false light (e.g., including covering foreign bodies with non-fluorescent materials) adds further work (potentially requiring specialized operators with specific training) and is not reproducible; in any case, interaction with the medical device can be difficult, especially during surgical procedures due to aseptic considerations.

[0008] WO-A-2013 / 096766 discloses a technique for imaging lesions in diagnostic applications. The mole boundary is located in a visible light image. The visible light image and the fluorescence image are aligned using a locator reference on both. Mole features are extracted from one or both images.

[0009] WO-A-2019 / 232473 discloses a technique for automatically detecting and characterizing micro-objects, such as cells or beads located within a microfluidic device. A neural network is used to process pixel data in an illuminated image to detect the micro-objects. Signals located within the corresponding boundaries of each detected micro-object in an unilluminated image, such as a fluorescence image, are used to measure the properties of the micro-object.

[0010] WO-A-2017 / 098010 discloses a technique for distinguishing between live-beads and blank-beads in DNA / RNA sequencing. The position of the beads is determined in a white light illumination image. The beads at their thus determined positions are classified based on the emission of electromagnetic radiation from fluorescent compounds. Summary of the Invention

[0011] In order to provide a basic understanding of this disclosure, a brief overview of this disclosure is given herein; however, the sole purpose of this overview is to introduce some concepts of this disclosure in a concise form as a prelude to its more detailed description below, and should not be construed as the identification of its key elements or the definition of its scope.

[0012] Generally speaking, this disclosure is based on the idea of ​​processing cold light images that are limited to information regions that can be identified in auxiliary images.

[0013] Specifically, one aspect provides a method for assisting a medical procedure. The method includes acquiring a cold light image (based on cold light) and an auxiliary image (based on an auxiliary light different from the cold light) of a field of view; the field of view includes a region of interest, which includes a target (containing a cold light material) for the medical procedure and one or more foreign objects. In the auxiliary image, an auxiliary information region representing the region of interest without foreign objects is identified based on its content, and a cold light information region is identified in the cold light image based on the auxiliary information region. The cold light image is then processed based on the cold light information region to facilitate the identification of a representation of the target within it.

[0014] On the other hand, a computer program for implementing the method is provided.

[0015] On the other hand, a corresponding computer program product is provided.

[0016] On the other hand, a computing device for performing the method is provided.

[0017] On the other hand, an imaging system including the computing device is provided.

[0018] On the other hand, it provides a corresponding medical procedure.

[0019] More specifically, one or more aspects of this disclosure are set forth in the independent claims and their advantageous features are set forth in the dependent claims, the wording of all claims being incorporated herein by reference verbatim (with any advantageous features provided by reference to any particular aspect, which is adapted as necessary to each of the other aspects). Attached Figure Description

[0020] The solutions disclosed herein, along with their additional features and advantages, will be best understood with reference to the following detailed description, given only by non-limiting indications and to be read in conjunction with the accompanying drawings (whereby, for simplicity, corresponding elements are indicated by the same or similar reference numerals without repeating their explanations, and the name of each entity is generally used to indicate both its type and its attributes, such as value, content, and representation). Specifically:

[0021] Figure 1 A schematic block diagram of an imaging system that can be used to implement solutions according to embodiments of the present disclosure is shown.

[0022] Figures 2A-2E Different examples of the application of solutions according to embodiments of this disclosure are shown.

[0023] Figure 3 The main software components that can be used to implement the solutions according to embodiments of this disclosure are shown, and

[0024] Figures 4A-4B and Figure 5 Different activity diagrams are shown that describe the activity flow related to the implementation of a solution according to an embodiment of the present disclosure. Detailed Implementation

[0025] For details, please refer to the following: Figure 1 A schematic block diagram of an imaging system 100 that can be used to implement solutions according to embodiments of the present disclosure is shown.

[0026] Imaging system 100 allows imaging of a corresponding field of view 103 (defined by a portion of the world within a stereoscopic angle perceptible to imaging system 100). Specifically, imaging system 100 is used for surgical applications (FGS, especially FGR) to assist surgeons. In this particular case, field of view 103 relates to patient 106 undergoing a surgical procedure, to which a fluorescent agent (e.g., suitable for accumulation in a tumor) has been applied. Field of view 103 includes a surgical cavity 109 (e.g., a small skin incision in minimally invasive surgery) that has been opened in patient 106 to expose the corresponding body part to be surgically operated on. Specifically, the body part exposed in surgical cavity 109 includes targets or other objects to which the surgeon must take action, such as tumor 112 to be removed. Field of view 103 generally includes one or more foreign objects (different from surgical cavity 109); for example, these foreign objects may include one or more surgical instruments 115 (such as scalpels), one or more hands 118 (such as a surgeon's hand), one or more surgical tools 121 (such as gauze), one or more body parts 124 (such as the skin around surgical cavity 109) and / or one or more background materials 125 (such as an operating table).

[0027] The imaging system 100 has an imaging probe 127 for acquiring images of the field of view 103 and a central unit 130 for controlling its operation.

[0028] Starting with imaging probe 127, it has an illumination unit (for illuminating the field of view 103) and an acquisition unit (for acquiring an image of the field of view 103), comprising the following components. In the illumination unit, excitation light source 133 and white light source 136 generate excitation light and white light, respectively. The excitation light has a wavelength and energy suitable for exciting fluorophores of a phosphor (such as near-infrared or NIR type), while white light is substantially colorless to the human eye (such as containing all wavelengths of the spectrum visible to the human eye at the same intensity). Corresponding transmission optics 139 and 142 transmit the excitation light and white light to the (same) field of view 103, respectively. In the acquisition unit, collection optics 145 collects light from the field of view 103 (in the apparent illumination geometry). The collected light includes fluorescence emitted by any fluorophores present in the field of view (illuminated by the excitation light). In practice, when fluorophores absorb excitation light, they enter an excited (electronic) state; the excited state is unstable, causing the fluorophore to quickly decay from the excited state to the ground (electronic) state, thus emitting fluorescence (at a characteristic wavelength, which is longer than the characteristic wavelength of the excitation light, because the energy in the excited state is dissipated as heat), the intensity of which depends on the amount of fluorophore being illuminated (as well as other factors, including the position of the fluorophore within the field of view and the body part). Furthermore, the collected light includes reflected light (in the visible spectrum), which is reflected by any object present in the field of view (illuminated by white light). Beam splitter 148 splits the collected light into two channels. For example, beam splitter 148 uses dichroic mirrors (or vice versa) to transmit and reflect the collected light at wavelengths above and below threshold wavelengths between the spectra of the reflected light and the fluorescence, respectively. In the (transmission) channel of beamsplitter 148, where fluorescence is defined by a portion of the light collected in its spectrum, emission filter 151 filters the fluorescence to remove any excitation light / white light (which may be reflected from the field of view) and ambient light (which may be generated by intrinsic fluorescence). A fluorescence camera 154 (e.g., an EMCCD type) receives the fluorescence from emission filter 151 and generates a corresponding fluorescence (digital) image representing the distribution of fluorophores in the field of view 103. In another (reflection) channel of beamsplitter 148, where reflected light is defined by a portion of the light collected in its spectrum, reflection, or photograph, camera 157 (e.g., a CCD type) receives the reflected light and generates a corresponding reflection (digital) image representing what is visible in the field of view 103.

[0029] Turning to the central unit 130, it comprises several units interconnected via a bus structure 160. Specifically, one or more microprocessors (μP) 163 provide the logical capabilities of the central unit 130. Non-volatile memory (ROM) 166 stores the basic code for booting the central unit 130, and volatile memory (RAM) 169 is used as working memory by the microprocessor 163. The central unit 130 is equipped with a mass storage device 172 (e.g., a solid-state drive or SSD) for storing programs and data. Furthermore, the central unit 130 includes multiple controllers 175 for peripheral devices, or input / output (I / O) units. Specifically, controller 175 controls the excitation light source 133, white light source 136, fluorescence camera 154, and reflection camera 157 of imaging probe 127; in addition, controller 175 controls other peripheral devices generally indicated by reference numeral 178, such as one or more monitors for displaying images, a keyboard for inputting commands, a trackball for moving pointers on monitor(s), a driver for reading / writing removable storage units (such as USB keys), and a network interface card (NIC) for connecting to a communication network (such as a LAN).

[0030] Now for reference Figures 2A-2E The following illustrates different examples of the application of solutions according to embodiments of the present disclosure.

[0031] from Figure 2A Beginning, a pair of corresponding reflective images 205R and fluorescence images 205F are shown. Reflective image 205R and fluorescence image 205F provide a coexisting representation of the same field of view 103 (in terms of reflected light and fluorescence, respectively). Specifically, the field of view 103 includes a surgical cavity 109 and some foreign objects, which in the example discussed include surgical instruments 115, two surgeons' hands 118, and surrounding skin 124; the foreign objects 115-124 may be arranged around the surgical cavity 109 (as in the case of surgical instruments 115, portions of surgeons' hands 118, and surrounding skin 124) or overlap with the surgical cavity 109 (as in the case of portions of surgeons' hands 118).

[0032] Go to Figure 2BIn the solution according to an embodiment of this disclosure, an information region 210Ri is identified in the reflected image 205R based on its content. Information region 210Ri represents the surgical cavity without foreign objects (in this example, surgical instruments, the surgeon's hand, and surrounding skin), followed by the information portion of interest in the field of view 103 (i.e., its region of interest, or ROI). The remaining portion of the reflected image 205R then defines a (reflected) non-information region 210Rn representing foreign objects, followed by non-information portions of interest in the field of view 103. This result is achieved, for example, through semantic segmentation techniques (such as the use of neural networks).

[0033] The identification of the information region 210Ri (and then the non-information region 210Rn) in the reflectance image 205R is transferred to the fluorescence image 205F. Specifically, the (fluorescent) information region 210Fi corresponding to the information region 210Ri is identified in the fluorescence image 205F. As a result, the remainder of the fluorescence image 205F defines the (fluorescent) non-information region 210Fn corresponding to the non-information region 210Rn.

[0034] As described in detail below, a processed image is now generated by processing a fluorescence image 205F limited to its information region 210Fi; therefore, the processing of the fluorescence image 205F is based solely on the (fluorescence) values ​​of the fluorescence image 205F within the information region, for example, based on their distribution (such as range, probability). This processing of the fluorescence image 205F is intended to facilitate the identification of tumor representations therein (e.g., through automatic scaling or thresholding of the information region 210Fi).

[0035] As a result, only the (informative) representation of the surgical cavity in the fluorescence image 205F can be considered, without considering the (non-informative) representation of the foreign body (around and / or overlapping it). This avoids (or at least substantially reduces) any adverse effects of the foreign body on the imaging of the surgical cavity. Specifically, the statistical distribution of fluorescence values ​​on which the processing of the fluorescence image 205F is based is now unbiased (because fluorescence values ​​in the non-informative region 210Fn do not contribute to this).

[0036] For example, Figure 2C Curves 215w and 215i are shown, representing the corresponding probability functions of fluorescence values ​​in the entire fluorescence image and only in its information region, respectively. Histograms approximate the fluorescence values ​​of probability functions 215w and 215i in a qualitative plot where fluorescence values ​​are plotted on the horizontal axis and their frequencies on the vertical axis. It can be seen that probability function 215i is much narrower than probability function 215w. Therefore, processing fluorescence images limited to their information regions benefits from this narrower probability distribution of fluorescence values.

[0037] Specifically, Figure 2D A processed image, referred to as the auto-scaled (fluorescence) image 220Fw, generated by automatically scaling the entire fluorescence image, is shown. The auto-scaled image 220Fw is obtained by applying a mapping function to all fluorescence values ​​of the fluorescence image to transform them from a (fluorescence) range given by the measurement resolution of the fluorescence to a (display) range given by the display dynamic range of the monitor used to display it (e.g., based on the logarithmic law to obtain an image with good contrast). The figure also shows a processed image, referred to as the auto-scaled (fluorescence) image 220Fi, generated by automatically scaling a fluorescence image limited to its information region 210Fi (and further by automatically scaling a fluorescence image limited to its non-information region 210Fn). Specifically, the auto-scaled image 220Fi is now obtained by separately applying corresponding mapping functions to the fluorescence values ​​in the information region 210Fi and the fluorescence values ​​in the non-information region 210Fn (where the fluorescence values ​​in the non-information region 210Fn are reduced by a scaling factor to mask their content). In the automatically scaled image 220Fw, the broad statistical distribution of its fluorescence values ​​limits the extent of fluorescence values ​​available within the display range used to map the fluorescence values ​​in the information region 210Fi (because their statistical distribution is narrower). This reduces the differences between fluorescence values ​​in the information region 210Fi, making it difficult (if not impossible) to identify portions with higher concentrations of fluorescent agent representing the tumor to be resected. Conversely, in the automatically scaled image 220Fi, the entire display range is available to map the fluorescence values ​​in the information region 210Fi. This increases the differences between fluorescence values ​​in the information region 210Fi, making the representation of tumor 112 more significant.

[0038] Go to Figure 2EThe figure illustrates different processed images generated by thresholding the entire fluorescence image, referred to as thresholded (fluorescence) images 225Fw. Specifically, thresholded image 225Fw is obtained by dividing the fluorescence image into (foreground) target segments and (background) non-target segments with fluorescence values ​​higher and lower than a threshold calculated based on the fluorescence values ​​(e.g., minimizing inter-class variance), respectively; then, the target segments representing the tumor (e.g., in color) are highlighted relative to the non-target segments representing the rest of the surgical cavity distinct from the tumor (e.g., in black and white). The figure also shows a processed image generated in the same manner by thresholding only the information region 210Fi, referred to as thresholded (fluorescence) image 225Fi. In the thresholded image 230Fw, the broad statistical distribution of its fluorescence values ​​increases the threshold (because its fluorescence values ​​are higher). This causes the fluorescence values ​​in the information region 210Fi to be classified into the non-target segments, thereby causing the tumor to disappear. Conversely, in the thresholded image 225Fi, the fluorescence values ​​in the information region 210Fi are correctly classified because the threshold is low (due to their narrower statistical distribution). This allows for the differentiation of tumor 112 within the information region 210Fi.

[0039] The solutions described above aid in the identification of tumors (or any other target). For example, the risk of overdetection (false positives) of tumors, and especially the risk of underdetection (false negatives), is significantly reduced. This avoids (or at least significantly reduces) excessive removal of healthy tissue, particularly incomplete removal of tumors; furthermore, it significantly improves the detection of tumor lesions. All of these have beneficial effects on the patient's health.

[0040] Now for reference Figure 3 The main software components that can be used to implement the solutions according to embodiments of this disclosure are shown.

[0041] All software components (programs and data) are collectively designated by reference numeral 300. Software components 300 are typically stored in mass storage and, during program execution, are loaded (at least partially) into the working memory of the central unit of the imaging system along with the operating system and other applications not directly related to the solutions disclosed herein (omitted in the figures for simplicity). Programs are initially installed into mass storage, for example, from removable storage units or from a communication network. In this respect, each program may be a module, segment, or portion of code, comprising one or more executable instructions for implementing specified logical functions.

[0042] A fluorescence driver 305 drives the fluorescence unit of the imaging system (including an excitation light source and a fluorescence camera) dedicated to acquiring fluorescence images of a field of view appropriately illuminated for this purpose. The fluorescence driver 305 accesses (in write mode) a fluorescence image library 310, which stores a sequence of fluorescence images acquired sequentially during the ongoing imaging process (to assist the corresponding surgical procedure). Each fluorescence image is defined by a bitmap comprising a cell matrix (e.g., having 512 rows and 512 columns), each cell storing the (fluorescence) value of a pixel, i.e., a basic image element corresponding to the (fluorescence) location in the field of view; each pixel value defines the brightness of the pixel as a function of the intensity of the fluorescence emitted at that location and the amount of phosphor present therein (e.g., from black when there is no phosphor to white as the amount of phosphor increases). Similarly, a reflection driver 315 drives the reflection unit of the imaging system (including a white light source and a reflection camera) dedicated to acquiring reflected images of a field of view appropriately illuminated for this purpose. The reflection driver 315 accesses (in write mode) the reflection image library 320, which stores a sequence of reflection images acquired sequentially during the same imaging process (synchronized with fluorescence images in the corresponding library 310). Each reflection image is defined by a bitmap comprising a cell matrix (of the same or different size relative to the reflection image), each cell storing the (reflection) value of a pixel corresponding to a (reflection) position in the field of view (the same or different relative to the fluorescence position); each pixel value defines the visible light (such as its RGB components) reflected from that position. The preparer 325 optionally preprocesses the reflection images to prepare them for subsequent identification of information regions therein. The preparer 325 accesses (in read mode) the reflection image library 320 and (optionally) the fluorescence image library 310, and it also accesses (in write mode) the prepared reflection image library 330. The prepared reflection image library 330 includes an entry for each reflection image in the corresponding library 320; the entry stores the corresponding prepared reflection image when the reflection image is suitable for identifying the information region, otherwise stores a null value. The prepared reflective images are formed by a matrix of cells with the same or different sizes as the reflective images, each cell storing the corresponding pixel (prepared) value. Segmenter 335 semantically segments the prepared reflective images into their informative and non-informative regions; each prepared reflective image is segmented based on its content and possibly also on the content of the corresponding fluorescence image (which, although semantically poor, can provide potentially useful additional information). Segmenter 335 accesses (in read mode) the prepared reflective image repository 330 and (optionally) the fluorescence image repository 310, and it accesses (in write mode) the reflective segmentation mask repository 340.The reflection segmentation mask library 340 includes entries for each of the prepared reflection image libraries 330; each entry stores the corresponding reflection segmentation mask for the prepared reflection image, or otherwise stores a null value. The reflection segmentation mask is formed by a matrix of cells having the same size as the prepared reflection image, each cell storing a label indicating the classification of the corresponding pixel; in the case of only two categories (one for information regions and one for non-information regions), the label is a binary value, for example, a declared (e.g., logical 1) binary value when the pixel belongs to an information region, and a dedeclared (e.g., logical 0) binary value when the pixel belongs to a non-information region. The equalizer 345 determines the optical properties related to the fluorescence of the material represented in the prepared reflection image, confined to its information region. The equalizer 345 accesses (in read mode) the prepared reflection image library 330 and the reflection segmentation mask library 340, and it accesses (in write mode) the reflection equalization map library 350. The reflectance equalization map library 350 includes entries for each of the prepared reflectance image libraries 330; each entry stores a corresponding reflectance equalization map for a prepared reflectance image, or otherwise stores a null value. The reflectance equalization map is formed by a matrix of cells having the same size as the prepared reflectance image, each cell storing (optical) values ​​(e.g., its reflection, absorption, etc.) related to the fluorescence of the material represented in the corresponding pixel. The adapter 355 optionally adapts the reflectance segmentation mask and the reflectance equalization map to the fluorescence image to equalize their sizes and synchronize them. The adapter 355 accesses (in read mode) the reflectance segmentation mask library 340, the reflectance equalization map library 350, and the fluorescence image library 310, and it accesses (in read / write mode) the fluorescence segmentation mask library 360 and the fluorescence equalization map library 365. The fluorescence segmentation mask library 360 includes a fluorescence segmentation mask corresponding to each fluorescence image in the library 310. The fluorescence segmentation mask is formed by a cell matrix of the same size as the fluorescence image, each cell storing a label for the corresponding pixel as described above (i.e., a pixel is declared or dedeclared when it belongs to an information region or a non-information region, respectively). The fluorescence equalization map library 365 includes a fluorescence equalization map corresponding to each fluorescence image in the library 310. The fluorescence equalization map is formed by a cell matrix of the same size as the fluorescence image, each cell storing the optical value of the corresponding pixel.

[0043] Processor 370 (post-processes) a fluorescence image limited to its information region to facilitate (e.g., by autoscaling and / or thresholding) the identification of tumor representations within it. Processor 370 accesses (in read mode) a fluorescence image repository 310, a fluorescence segmentation mask repository 360, and a fluorescence equalization map repository 365, and it accesses (in write mode) a processed fluorescence image repository 375. The processed fluorescence image repository 375 includes a processed fluorescence image corresponding to each fluorescence image in repository 310; for example, the processed fluorescence image is an autoscaled fluorescence image (in the case of autoscaling) or a thresholded fluorescence image (in the case of thresholding). The processed fluorescence image is formed by a cell matrix having the same size as the fluorescence image, with each cell storing the corresponding pixel (processed, i.e., autoscaled / thresholded) value. Visualizer 380 generates an output image based on the processed fluorescence image for its visualization. Visualizer 380 accesses (in read mode) the processed fluorescence image library 375 and (optionally) the fluorescence mask library 360 and the reflectance image library 320, and it accesses (in write mode) the output image library 385. The output image library 385 includes output images corresponding to each processed fluorescence image in the library 375. For example, the output image is equal to a single processed fluorescence image, or equal to a processed fluorescence image superimposed on its corresponding reflectance image. Monitor driver 390 drives the monitor of the imaging system to display the output images (essentially in real-time during the surgical procedure). Monitor driver 390 accesses (in read mode) the output image library 385.

[0044] Now for reference Figures 4A-4B and Figure 5 The diagram illustrates different activity diagrams describing the activity flow related to the implementation of a solution according to embodiments of the present disclosure. In this respect, each block may correspond to one or more executable instructions for implementing a specified logical function on a corresponding computing device.

[0045] from Figures 4A-4B Initially, their activity diagrams represent exemplary procedures that can be used to image a patient using method 400. During surgical procedures on the patient, this process is performed on the central unit of the imaging system. Solutions according to embodiments of this disclosure (where the identification of information regions in a reflected image is transferred to a corresponding fluorescence image to facilitate tumor identification therein) can be applied indiscriminately (always) or selectively (e.g., by activating / deactivating it in response to a corresponding command, such as pressing a dedicated button on the imaging system).

[0046] Before (or even days before) a surgical procedure, a healthcare operator (e.g., a nurse) administers a fluorescent agent to the patient. Fluorescent agents (e.g., indocyanine green, methylene blue, etc.) are adapted to reach a specific (biological) target, such as the tumor to be removed. This can be achieved by using non-targeted fluorescent agents (adapted to accumulate in the target without any specific interaction with it, such as through passive accumulation) or targeted fluorescent agents (adapted to attach to the target through specific interactions, such as by incorporating target-specific ligands into the formulation of the fluorescent agent, for example based on chemical binding properties and / or physical structures suitable for interaction with different tissues, vascular properties, metabolic characteristics, etc.). A dose of the fluorescent agent is administered intravenously to the patient (using a syringe); as a result, the fluorescent agent circulates within the patient's vascular system until it reaches and binds to the tumor; conversely, any remaining (unbound) fluorescent agent is cleared from the blood pool (according to the corresponding half-life). After a waiting period (e.g., from a few minutes to 24–72 hours) allowing the fluorescent agent to accumulate in the tumor and be washed out from other parts of the patient's body, the surgery can begin. At this point, the operator places the imaging probe near the patient area where the surgeon has opened the surgical cavity; then the operator inputs the start command into the imaging system (e.g., using its keyboard).

[0047] In response, the imaging process begins by moving from black starting circle 402 to box 404. At this point, the fluorescence driver and reflection driver respectively activate the excitation light source and white light source to illuminate the field of view. The activity flow then branches into two concurrently executed operations. Specifically, the fluorescence driver acquires a (new) fluorescence image at box 406 and adds it to the corresponding memory. Simultaneously, the reflection driver acquires a (new) reflection image at box 408 and adds it to the corresponding memory. In this way, the fluorescence image and the reflection image are acquired essentially simultaneously, and they provide different representations of the same field of view that are spatially coherent (i.e., there is a predictable correlation between their pixels, up to perfect consistency) (in terms of fluorescence and visible light, respectively).

[0048] The activity flow converges again from boxes 406 and 408 at box 410, where the preparer retrieves the reflected image just added to the corresponding store and optionally preprocesses it for the next recognition of the information region within it. For example, the preparer can verify whether the reflected image is suitable for recognizing the information region. To do this, the mean and / or variance of its pixel values ​​can be calculated. If the mean is (possibly strictly) below a (darkness) threshold (meaning the reflected image is too dark) and / or if the variance is (possibly strictly) above a (blur) threshold (meaning the reflected image is too blurry), the quality of the reflected image is considered unacceptable for meaningful recognition of the information region; in this case, the preparer ignores the reflected image by adding null values ​​to the prepared reflected image store. Conversely (meaning the quality of the reflected image is acceptable, and recognizing the information region within it is feasible), the preparer can apply one or more filters to further improve the quality of the reflected image (e.g., normalize color, reduce noise, correct illumination, reduce distortion, remove reflections, etc.). Specifically, if the average value (possibly strictly) is above a darkness threshold but below a higher (brightness) threshold (meaning the reflected image is not very bright), such as equal to 1.2–1.5, the preparer can apply histogram equalization to the reflected image (by expanding the most frequent pixel values ​​to obtain a substantially flat histogram); in fact, experimental results show that histogram equalization improves performance in this case, whereas it might otherwise degrade performance. Alternatively, the preparer can shrink the reflected image to reduce computational complexity (e.g., using low-pass filtering followed by subsampling). Alternatively, the preparer can group the pixels of the reflected image into substantially homogeneous groups, each group represented by a group value based on the corresponding pixel values, to simplify the identification of informative regions (e.g., by applying clustering, graph-based, random walk, watershed edge detection, and similar algorithms). Alternatively, the preparer can apply motion compensation algorithms (to align the reflected image with the fluorescence image) and / or warping algorithms (to correct distortion of the reflected image relative to the fluorescence image). In any case, the preparer then adds the prepared reflection image (which may be equal to the corresponding reflection image) to the corresponding storage.

[0049] The activity flow branches at box 412 based on the content of the entry just added to the prepared reflection image repository. If the entry contains a (prepared) reflection image, the segmenter retrieves that reflection image from the corresponding repository at box 414 for its semantic segmentation. In computer vision, semantic segmentation is a specific type of segmentation (generally aimed at dividing an image into disjoint parts or segments with fundamentally homogeneous properties), where segments represent entities belonging to different categories with corresponding meanings (i.e., concepts abstracting the common properties of multiple instances). In this specific case, semantic segmentation aims to divide the reflection image into informational regions representing surgical cavities without foreign objects and non-informational regions representing foreign objects (i.e., surgical instruments, hands, surgical tools, surrounding body parts, and / or background material). The activity flow then branches at box 416, depending on the segmenter's implementation. Specifically, boxes 418-422 are executed when the segmenter is based on a classification algorithm, while box 424 is executed when the segmenter is based on deep learning techniques.

[0050] Now referring to box 418 (classification algorithm), the segmenter performs a feature extraction step to extract one or more features from the reflectance image and possibly also from the corresponding fluorescence image (pre-determined to be best suited for this purpose, as described below); each feature is a (measurable) property representing a unique characteristic of the reflectance / fluorescence image. Examples of these features include saturation, hue, brightness, histogram of oriented gradients (HOG), variance, bag of visions (BOV), scale-invariant feature transform (SIFT), and so on. More specifically, the segmenter computes one or more feature maps; each feature map is formed by a matrix of cells of the same size as the reflectance / fluorescence image, with each cell storing the (feature) value of the corresponding feature. For this purpose, the segment applies a corresponding filter to the reflectance / fluorescence image (e.g., smoothing such as Gaussian blur, Kuwhara, anisotropic diffusion, etc.; statistics such as mean, median, entropy, etc.; edge detectors such as Sobel, Prewitt, Canny, etc.; derivatives, Hessian, Laplacian, etc.); each filter computes a feature value for each location based on the corresponding pixel value, possibly considering the pixel values ​​of its neighboring pixels. At box 420, the segmenter computes a reflection segmentation mask corresponding to the reflection image by applying a specific classification algorithm to the feature map and then adds it to the corresponding storage. For example, the classification algorithm is a Conditional Random Field (CRF) algorithm. Essentially, a CRF algorithm computes a label for each pixel through an inference step that determines a label value that maximizes the posterior probability that the pixel belongs to the corresponding class. The posterior probability is based on the node (or unary) potential that depends only on the pixel's feature values ​​and the edge (or paired) potential that considers its neighboring pixels (their labels are used to smooth the transition between segments, or their feature values ​​are used to model similarity). At box 422, the segmenter optionally enhances the reflection segmentation mask thus obtained. For example, the segmenter may perform a hole-filling step, where any disconnected portions of non-informative regions (i.e., completely surrounded by informative regions) are assigned to informative regions (assuming no foreign objects can be completely surrounded by the cavity). Alternatively or additionally, the segmenter may perform one or more smoothing steps to remove isolated misclassified pixels (such as by applying erosion, dilation, box-filtered convolution, and similar algorithms).

[0051] Conversely, in reference box 424 (deep learning), the segmenter is an (artificial) neural network, such as a U-Net (appropriately trained for this purpose, as described below). Essentially, deep learning is a specific type of machine learning (used to perform a specific task, in this case, semantically segmenting a reflection image, without using explicit instructions, but automatically inferring how to do it from examples) that is based on neural networks. A neural network is a data processing system that closely resembles the operation of the human brain. A neural network consists of basic processing units (neurons) that perform operations based on corresponding weights; nodes are connected via unidirectional channels (synapses) that transmit data between them. Neurons are organized into layers that perform different operations, always including an input layer for receiving input data and an output layer for providing output data (in this case, possibly with a corresponding fluorescence image and a reflection image of a corresponding reflection segmentation mask, respectively). A deep neural network (DNN) is a specific type of neural network with one or more (hidden) layers between the input and output layers. A convolutional neural network (CNN) is a specific type of deep neural network in which one or more of its layers perform (cross-convolution) operations. Specifically, a CNN comprises one or more convolutional layers that compute corresponding feature maps, followed by one or more pooling layers that reduce the resolution of the feature maps; then, one or more fully connected layers segment the fluorescence image based on these (reduced) feature maps. A U-Net is a specific type of convolutional neural network where a contraction path (formed by convolutional and pooling layers) is followed by an expansion path; conversely, the expansion path comprises one or more upsampling layers that increase the resolution of the feature maps, followed by one or more convolutional layers that assemble them, without any fully connected layers (where the expansion path is substantially symmetrical to the contraction path, thus providing a U-shaped architecture). In this case, the segmenter (receiving the reflection / fluorescence image) directly generates a reflection segmentation mask and adds it to the corresponding storage.

[0052] In both cases, the activity flow merges again from box 422 or box 424 at box 426. At this point, the equalizer retrieves the reflection segmentation mask just added to its store and the corresponding (prepared) reflection image from its store to determine the optical properties of the reflection image confined to its information region. For this purpose, the equalizer considers each pixel of the reflection image; if the corresponding label in the reflection segmentation mask is declared (meaning the pixel belongs to an information region), the equalizer determines the type of biological material represented by the corresponding pixel value (e.g., blood, muscle, fat, etc., depending on the color of the pixel value) and then adds its optical value (e.g., a range from 0 to 1, depending on the type of biological material and the brightness of the pixel value) to the corresponding cell in the reflection equalization map; conversely, if the corresponding label in the reflection segmentation mask is undeclared (meaning the pixel belongs to a non-information region), the equalizer adds a null value to the corresponding cell in the reflection equalization map. The adapter retrieves the reflection segmentation mask and reflection equalization map just added to the corresponding store at box 428 and optionally adapts them to the corresponding fluorescence image (from the corresponding store). For example, when the reflection segmentation mask and the reflection equalization map have different sizes (e.g., by low-pass filtering followed by subsampling, or by interpolation followed by low-pass filtering, respectively), the adapter can scale down / up the reflection segmentation mask and the reflection equalization map to fit the fluorescence image. In any case, the adapter adds the fluorescence segmentation mask and the fluorescence equalization map (equal to the reflection segmentation mask and the reflection equalization map that may be adapted to the fluorescence image, respectively) to the corresponding libraries.

[0053] Returning to reference box 412, if the entry just added to the prepared reflectance image library contains null values, the process proceeds to box 430 instead. In this case, the adapter estimates the fluorescence segmentation mask and fluorescence equalization map corresponding to the lost (prepared) reflectance image, respectively, based on one or more previous fluorescence segmentation masks and fluorescence equalization maps (extracted from the corresponding libraries). For example, each of the fluorescence segmentation mask and fluorescence equalization map is simply set to be equal to the previous one, or calculated by interpolating the previous two or more. As described above, the adapter adds the fluorescence segmentation mask and fluorescence equalization map thus obtained to the corresponding libraries.

[0054] In any case, the activity flow merges again from either box 428 or box 430 at box 432. At this point, the processor retrieves the fluorescence segmentation mask and fluorescence equalization map, along with the corresponding fluorescence image, that were just added to its storage for (post)processing of the fluorescence image, which is confined to its information region (and possibly also its non-information region). For example, firstly, the processor may optionally equalize the fluorescence image (constrained to its information region) based on the corresponding optical properties of the reflectance image. To do this, the equalizer considers each pixel of the fluorescence image; if a corresponding label in the fluorescence segmentation mask is declared (meaning the pixel belongs to the information region), the equalizer updates its pixel value based on the corresponding optical value in the fluorescence equalization map (e.g., by increasing the pixel value when the optical value indicates that the corresponding biological material (such as blood) significantly obscures the fluorescence). In this way, the influence of different biological materials on the acquired fluorescence can be compensated; specifically, confining this operation to the information region only avoids the potential adverse effects of foreign objects on the results. The activity flow branches at box 434 according to the type of processing to be applied to the (possibly equalized) fluorescence image. Specifically, in the case of autoscaling, boxes 436-446 are executed, while in the case of thresholding, boxes 448-464 are executed. In both cases, the activity flow is merged again at box 466.

[0055] Referring now to box 436 (auto-scaling), the processor determines the (information) fluorescence range of the information region as the difference between its highest and lowest pixel values, and the (non-information) fluorescence range of the non-information region as the difference between its highest and lowest pixel values. The monitor's information fluorescence range and a predefined display range (retrieved from the corresponding configuration variables) are used as parameters to a predefined parametric function (e.g., logarithmic type) to obtain the corresponding information mapping function; similarly, the non-information fluorescence range and display range are used as parameters to a predefined parametric function (same as above or different, such as by adding a scaling factor to mask the content of the non-information region) to obtain the corresponding non-information mapping function. A loop then enters to auto-scale the information and non-information regions separately. The loop begins at box 438, where the processor considers the (current) pixels of the fluorescence image (starting from the first pixel in an arbitrary random order). The process branches at box 440 based on the corresponding label in the fluorescence segmentation mask. If a label is declared (meaning the pixel belongs to an information region), the processor transforms the corresponding pixel value at box 442 by applying an information mapping function and adds it to the same cell in the (new) auto-scaled fluorescence image (in the corresponding temporary variable). Conversely, if a label is dedeclared (meaning the pixel belongs to a non-information region), the processor transforms the corresponding pixel value at box 444 by applying a non-information mapping function and adds it to the same cell in the auto-scaled fluorescence image. In both cases, the processor verifies at box 446 whether the last pixel of the fluorescence image has been processed. If not, the process returns to box 438 to repeat the same operation for the next pixel. Otherwise (once all pixels have been processed), the loop exits by going down to box 466.

[0056] Conversely, referring to box 448 (thresholding), the processor determines the threshold using only the pixel values ​​of the information region (e.g., by applying the Otsu algorithm to them). A loop then begins to threshold the information region. The loop starts at box 450, where the processor considers the (current) pixels of the fluorescence image (starting from the first pixel in an arbitrary random order). The process branches at box 452 based on the corresponding label in the fluorescence segmentation mask. If the label is declared (meaning the pixel belongs to an information region), the processor compares the corresponding pixel value to the threshold at box 454. If the pixel value is (possibly strictly) above the threshold (i.e., it belongs to the target segment), the processor copies it to the same cell in the (new) thresholded fluorescence image (in the corresponding temporary variable) at box 456. Conversely, if the pixel value is (possibly strictly) below the threshold (i.e., it belongs to a non-target segment), the processor resets the pixel value in the same cell of the thresholded fluorescence image to zero at box 458 (to mask it). If the label is dedeclared (meaning the pixel belongs to a non-information region), the same point is reached from box 452. In any case, the process continues from box 456 or box 458 to box 460. At this point, the processor verifies whether the last pixel of the fluorescence image has been processed. If not, the process returns to box 450 to repeat the same operation for the next pixel. Conversely (once all pixels have been processed), the loop exits by going down to box 462. Alternatively, the processor creates a thresholding mask (formed by a matrix of cells of the same size as the fluorescence image, each cell storing a flag); for each pixel value of the fluorescence image, if the corresponding label in the fluorescence segmentation mask is declared and the pixel value is (possibly strictly) above the threshold, the processor declares the corresponding flag (e.g., a value of 1); otherwise, the processor dedeclares the corresponding flag (e.g., a value of 0). Now referring to box 462, the processor can further process the thresholded fluorescence image thus obtained. For example, the processor calculates one or more (target) statistical parameters of the target segment and one or more (non-target) statistical parameters of the non-target segment of the thresholded fluorescence image (e.g., their mean and standard deviation). To this end, considering the thresholding mask case (with similar considerations applied in other cases), the processor considers each pixel of the fluorescence image; if the corresponding label in the thresholding mask is declared (meaning the pixel belongs to the target segment), the processor uses the corresponding pixel value to increment the calculation of the target segment statistics. Conversely, if the corresponding label in the thresholding mask is dedeclared and the label in the fluorescence segmentation mask is declared (meaning the pixel belongs to the non-target segment), the processor uses the corresponding pixel value to increment the calculation of the non-target segment statistics. At box 464, the processor updates the pixel values ​​of the target segment based on the target segment statistics, the non-target segment statistics, or both.To this end, the processor considers each pixel of the thresholded fluorescence image; for example, if a corresponding flag in the thresholded mask is declared (meaning the pixel belongs to the target segment), the processor subtracts the average of the non-target segments from the corresponding pixel value, divides the result or (raw) pixel value by the standard deviation of the target segment, by the standard deviation of the non-target segment, their combinations (such as their sum, difference, average, etc.), and so on. This process then proceeds to box 466.

[0057] Referring now to box 466, the processor adds the processed fluorescence image thus obtained to the corresponding repository (and adds any possible corresponding thresholding masks to another repository). The visualizer retrieves the processed fluorescence images just added to the corresponding repositories at box 468, and optionally retrieves the corresponding fluorescence segmentation masks and reflectance images (and any possible corresponding thresholding masks) from their repositories. The visualizer generates a corresponding output image (based on the processed fluorescence image) and adds it to the corresponding repository. For example, the visualizer can simply set the output image to be equal to the processed fluorescence image itself (in the case of a thresholding mask, the same result is obtained by masking the pixel values ​​of the fluorescence image whose corresponding flag is declaring in its thresholding mask). Alternatively or additionally, the visualizer (after rescaling the processed fluorescence or reflectance images to make them equal in size if necessary) can generate a combined image given by the pixel values ​​of the processed fluorescence and reflectance images in the information and non-information regions, respectively, or an overlapping image given by the pixel values ​​of the processed fluorescence and reflectance images that are higher than zero (or whose flags in their thresholding masks are declared). In any case, the monitor driver displays the output image that has just been added to the corresponding storage library at box 470. In this way, the output image is displayed essentially in real time along with the acquisition of the corresponding fluorescence / reflection image, except for a short delay due to their generation.

[0058] Referring now to frame 472, if the imaging process is still in progress, the activity flow returns before frames 406-408 to continuously repeat the same operation. Conversely, if the imaging process has ended, as indicated by an operator (e.g., using their keyboard) entering an end command for the imaging system, the process ends at the concentric white / black stop circle 474 (after the excitation and white light sources are turned off via the corresponding drivers).

[0059] Go to Figure 5The activity diagram illustrates an exemplary process that can be used to configure the segmenter using method 500 (during the development of the imaging system and possibly during its next validation). This process is executed on a configuration (computing) system, such as a personal computer (which, as described above, includes one or more microprocessors, non-volatile memory, volatile memory, mass storage, and controllers for its peripherals). For this purpose, the configuration system includes the following software components: A configurator for configuring the segmenter. The configurator accesses (in read / write mode) a reflectance image library storing (image) sequences of multiple (reference) reflectance images, and it accesses (in read / write mode) a reflectance segmentation mask library storing corresponding (sample) reflectance segmentation masks.

[0060] The process begins at the black starting circle 503, then proceeds to box 506, where the configurator uploads (e.g., via a removable storage unit or communication network) image sequences of multiple reflective images acquired via the corresponding imaging system during different (sample) surgical procedures as described above (e.g., each of dozens of reflective images has hundreds of image sequences), and adds them to the corresponding storage. Continuing to box 509, each reflective image is manually segmented into its (reflective) informative region and (reflective) non-informative region. For example, this is achieved through a semi-automatic method, where initial segmentation is performed automatically (e.g., by applying the Simple Interactive Object Extraction (SIOX) algorithm), and the obtained results are manually refined to correct for possible errors. The reflective segmentation mask thus obtained is then added to the corresponding storage.

[0061] The configurator selects a training set at box 512, which is formed from a subset of available reflection images and corresponding reflection segmentation masks (image / mask) pairs. For example, the training set can be defined by randomly and indiscriminately selecting from all available image / mask pairs (so that the longer the image sequence, the higher its sampling frequency) or by making them homogeneous across the image sequence (so that the same number of image / mask pairs are provided for each image sequence). The activity flow then branches at box 515 depending on the segmenter's implementation. Specifically, boxes 518-521 are executed when the segmenter is based on a classification algorithm, and box 524 is executed when the segmenter is based on a deep learning technique.

[0062] Referring now to box 518 (classification algorithm), the configurator performs a feature selection step to determine an optimized feature set from a large number of possible (candidate) features to improve the segmenter's performance. For example, the feature selection step is based on a wrapper method, where iterative optimization of the classification algorithm is used to determine the optimized set. To this end, a brute-force approach is first applied to initialize the optimized set. To make the computational complexity feasible in practice, the initialization of the optimized set is limited to a maximum size of a few units (e.g., ≤3), and a simplified version of the classification algorithm is used (e.g., in the case of the CRF algorithm, it is limited to a node potential with a default model, such as a Naive Bayes-based algorithm). To this end, the configurator considers each (feature) combination formed by a number of candidate features at most equal to the maximum size. For each feature combination, the configurator causes the segmenter to compute a reflection segmentation mask corresponding to the reflection image of the training set by applying the classification algorithm with that feature combination. The configurator computes a quality index indicating the quality of the segmentation provided by the feature combination. To this end, the configurator calculates a similarity index that measures the similarity between each (calculated) reflection segmentation mask thus obtained and its corresponding (reference) reflection segmentation mask in the training set; for example, the similarity index is derived from... The coefficient is defined as twice the number of pixels with the same label in the calculated / reference reflection segmentation mask divided by the total number of pixels (from 0 to 1 in ascending order of similarity), or as defined by indices such as Jaccard, Bray–Curtis, Czekanowski, Steinhaus, Pielou, and Hellinger. The segmenter then calculates a quality index as the average of the similarity indices of all calculated reflection segmentation masks relative to their corresponding reference reflection segmentation masks. The optimization set is initialized to provide the feature combination with the highest quality index. The configurator then applies a stepwise approach to expand the optimization set. To do this, the configurator considers each additional candidate feature not yet included in the optimization set. For each additional (candidate) feature, the configurator causes the segmenter to compute a reflection segmentation mask corresponding to the reflection image in the training set by applying a classification algorithm with the features from the optimization set plus that additional feature, and computes the corresponding quality index as described above. If the (best) additional feature that provides the highest quality index when added to the optimization set results in a significant improvement (e.g., the difference between the quality index of the optimization set plus the best additional feature and the quality index of the (original) optimization set is (potentially strictly) higher than a minimum, such as 5-10%), the configurator adds the best additional feature to the optimization set. This process is repeated until an acceptable quality index is obtained, the best additional feature does not provide any significant improvement, or the optimization set reaches its maximum allowed size (e.g., 10-15). The configurator optionally selects one or more operational parameters of the classification algorithm that optimizes the segmenter's performance at box 521. For example, in the case of the CRF algorithm, this involves selecting optimized node models and optimized edge models, respectively, for calculating node and edge potentials, and selecting optimized values ​​for their (node) model parameters and (edge) model parameters, respectively. To make the computational complexity of the operation practically feasible, the selection of optimized node / edge models is performed using an empirical method independent of the feature selection step (i.e., by using the optimization set determined above). For example, first, the configurator selects an optimal node model from multiple possible (candidate) node models (e.g., based on Naive Bayes, Gaussian mixture models, k-nearest neighbors, artificial neural networks, support vector machines, and similar algorithms) by keeping the edge model fixed at a default model (e.g., the Potts model) and using default values ​​for the node / edge model parameters. To this end, for each candidate node model, the configurator causes the segmenter to compute a reflection segmentation mask corresponding to the reflection images of the training set by applying a classification algorithm with that candidate node model, and computes the corresponding quality index as described above. The configurator sets the optimal node model as the candidate node model that provides the highest quality index.Subsequently, the configurator optimizes the edge model (e.g., Potts-based, contrast-sensitive Potts, contrast-sensitive Potts models with prior probabilities, and similar algorithms) from multiple possible (candidate) edge models using default values ​​of the node / edge model parameters. For each candidate edge model, the configurator causes the segmenter to compute a reflection segmentation mask corresponding to the reflection image of the training set by applying the classification algorithm with that candidate edge model, and calculates the corresponding quality index as described above. The configurator sets the optimized edge model as the candidate edge model that provides the highest quality index. Finally, the configurator searches for optimized values ​​for the node / edge model parameters against the optimized node and edge models determined above. To do this, the configurator causes the segmenter to compute a reflection segmentation mask corresponding to the reflection image of the training set by applying the classification algorithm with the optimized node / edge model and changing its model parameters, and calculates the corresponding quality index as described above. The operation is driven by an optimization method (e.g., using the Powell search algorithm) until an acceptable quality index is obtained.

[0063] Conversely, referring to box 524 (Deep Learning), the configurator performs training steps on the segmenter's neural network using the training set to find (optimized) values ​​for its weights to optimize the segmenter's performance. To make the computational complexity feasible in practice, the training steps are based on an iterative process, such as stochastic gradient descent (SGD). To this end, the configurator initializes the neural network's weights at the beginning (e.g., randomly). The configurator inputs the reflection images from the training set into the neural network to obtain corresponding reflection segmentation masks and calculates the corresponding quality index as described above. The configurator determines the weight changes that should improve the neural network's performance; specifically, in the SGD algorithm, the direction and amount of the changes are given by the gradient of the error function relative to the weights, approximated by backpropagation. The same operation is repeated until an acceptable quality index is obtained or the weight changes do not provide any significant improvement (meaning a minimum, at least a local minimum, or a flat region of the error function has been found). The weights can be changed in an iterative mode (after obtaining each reflection segmentation mask) or in a batch mode (after obtaining all reflection segmentation masks). In any case, the weights are altered by adding random noise, and / or the training steps are repeated starting from one or more different initializations of the neural network to find different (and possibly better) local minima and to distinguish flat regions of the error function. In this way, the features to be used for segmenting the reflection image (implicitly defined by the weights) are automatically determined during the training steps without requiring any explicit selection of them.

[0064] In both cases, the activity flow merges again from either box 521 or box 524 at box 527. Here, the configurator selects a test set formed from a portion of the image / mask pairs, defined by randomly and indiscriminately sampling them from all available image / mask pairs or homogeneously sampling them across the image sequence. At box 530, the configurator causes the segmenter thus obtained to compute a reflection segmentation mask corresponding to the reflection image of the test set, and calculates the corresponding quality index as described above. The activity flow branches at box 533 based on the quality index. If the quality index is (possibly strictly) below an acceptable value, this means the segmenter's generalization ability (from its configuration based on the training set to the test set) is too poor; in this case, the process returns to box 512 to repeat the same operation on a different training set (or to box 506 to add an image sequence of reflection images, not shown in the figure). Conversely, if the quality index is (possibly strictly) above an acceptable value, this means the segmenter's generalization ability is satisfactory; in this case, the configurator accepts the segmenter's configuration at box 536 and deploys it to a batch of imaging systems. Then, the process ends at point 539, the concentric white / black stopping circle.

[0065] Revise

[0066] Naturally, those skilled in the art can apply numerous logical and / or physical modifications and alterations to this disclosure to meet local and specific requirements. More specifically, although this disclosure has been described with reference to one or more embodiments thereof to a certain degree of specificity, it should be understood that various omissions, substitutions, and changes in form and detail, as well as in other embodiments, are possible. Specifically, different embodiments of this disclosure may be implemented even without the specific details (such as numerical values) set forth in the foregoing description to provide a more thorough understanding thereto; rather, well-known features may be omitted or simplified in order not to obscure this description with unnecessary detail. Furthermore, specific elements and / or method steps explicitly intended to be described in connection with any embodiment of this disclosure may be incorporated into any other embodiment as a matter of general design choice. In addition, items presented in the same group and in different embodiments, examples, or alternatives should not be construed as being factually equivalent to each other (but they are separate and autonomous entities). In any case, each numerical value should be interpreted as modified according to applicable tolerances; specifically, unless otherwise indicated, the terms “substantially,” “approximately,” “about,” etc., should be understood as within 10%, preferably 5%, more preferably 1%. Furthermore, each numerical range should explicitly specify any possible numbers along the continuum within that range (including its endpoint). Ordinal numbers or other qualifiers serve only as labels to distinguish elements with the same name, but do not imply any priority, order, or sequence. Terms such as include, contain, have, contain, and relate to should be intended to have an open, non-exhaustive meaning (i.e., not limited to the items described); terms such as based on, depend on, according to, and its function should be intended to represent a non-exclusive relationship (i.e., relating to possible other variables); the term "one" or "a" should be intended to represent one or more items (unless otherwise expressly indicated); and the term used for a component (or any component plus function) should be intended to represent any structure adapted or configured to perform the relevant function.

[0067] For example, an embodiment provides a method for assisting in medical procedures on a patient. However, this method can be used to assist in any medical procedure (e.g., surgical procedure, diagnostic procedure, treatment procedure, etc.) on any patient (e.g., human, animal, etc.); furthermore, the corresponding steps can be performed in any manner (e.g., continuously during a medical procedure, on request, etc.). In any case, although the method may assist the physician's task, it remains a data processing method that only provides information that may be helpful to the physician, but has a strictly defined medical function that is always performed by the physician himself / herself.

[0068] In this embodiment, the method includes the following steps under the control of a computing device. However, the computing device can be of any type (see below).

[0069] In one embodiment, the method includes acquiring a cold light image of the field of view (by a computing device). However, the cold light image can be acquired in any manner (e.g., at any frequency, with any excitation light, directly by controlling any acquisition unit, transmitted with a removable storage unit, uploaded via a network, etc.).

[0070] In this embodiment, the field of view includes the patient's area of ​​interest (for medical procedures). However, the area of ​​interest can be of any type (e.g., the surgical cavity of a surgical procedure, the lumen of an endoscopic procedure, an open type accessed via a hollow or a closed type accessed via an incision, etc.).

[0071] In this embodiment, the region of concern includes at least one target of the medical procedure. However, the target can have any number and any type (e.g., a lesion to be removed, identified, monitored, or treated, such as a tumor, polyp, inflammation, thrombosis, etc.; a body part to be repaired, such as a bleeding vessel to be cauterized; a narrow esophagus to be widened, etc.; a structure surrounding any item that the physician should take action on, etc.).

[0072] In this embodiment, the target contains a luminescent material. However, the luminescent material can be of any extrinsic / intrinsic or exogenous / endogenous type (e.g., any luminescent agent, any natural luminescent component, based on any luminescent phenomenon, such as fluorescence, phosphorescence, chemiluminescence, bioluminescence, induced Raman radiation, etc.).

[0073] In this embodiment, the field of view includes one or more foreign objects that are different from the region of interest. However, the foreign objects can be any number and any type (e.g., instruments, hands, tools, body parts, background material, etc.).

[0074] In an embodiment, the cold light image includes a plurality of cold light values ​​representing the cold light emitted by the cold light material at a corresponding cold light location in the field of view. However, the cold light image can have any size and shape (from the entire matrix to one or more of it), and it can include cold light values ​​of any type and at any cold light location (e.g., gray levels or color values ​​in RBG, YcBcr, HSL, CIE-L*a*b, Lab color, etc., representations for pixels, voxels, pixel groups, voxel groups, etc.); the cold light can be of any type (e.g., NIR, infrared (IR), visible light, etc.), and it can be emitted in any manner (e.g., in response to a corresponding excitation light, or more generally, in response to any other excitation different from heating).

[0075] In an embodiment, the method includes acquiring an auxiliary image of the field of view (by a computing device). However, the auxiliary image can be acquired in any manner (e.g., identical or different from the cold light image, concurrent with or shortly consecutive to the cold light image, etc.).

[0076] In an embodiment, the auxiliary image includes a plurality of auxiliary values ​​representing auxiliary light (different from cold light) received from a corresponding auxiliary position in the field of view. However, the auxiliary image can have any size and shape, and it can include auxiliary values ​​of any type and for any auxiliary position (same or different relative to the cold light image); the auxiliary light can be any type of cold light different from the cold light image (e.g., visible light, IR light, ultraviolet (UV) light, other cold light of different wavelengths, etc.).

[0077] In an embodiment, the method includes (by a computing device) identifying an auxiliary information region of an auxiliary image that represents a region of interest free of foreign objects, based on the content of the auxiliary image. However, the auxiliary information region can be of any type (e.g., a single region, one or more non-intersecting regions, defined by a corresponding mask, or directly in the auxiliary image, etc.), and it can be identified in any manner (e.g., by semantically / non-semantically segmenting the auxiliary image into informational and non-informational regions, by searching for informational regions in the auxiliary image, etc.).

[0078] In an embodiment, the method includes (by a computing device) identifying cold light information regions of a cold light image corresponding to auxiliary information regions. However, cold light information regions can be identified in any manner (e.g., by directly transferring the identification of auxiliary information regions or by any adaptation, etc.). Furthermore, this operation can be performed indiscriminately, or it can be conditional on the quality of the identification of auxiliary information regions; for example, if the quality metric does not reach a corresponding threshold, a quality metric (for the entire process or its steps) can be calculated and all cold light locations can be assigned to cold light information regions.

[0079] In an embodiment, the method includes generating a processed cold light image (by a computing device). However, the processed cold light image can be of any type (e.g., an automatically scaled fluorescence image, a thresholded fluorescence image, a segmented fluorescence image, etc.).

[0080] In this embodiment, a processed cold light image is generated by processing a cold light image limited to the cold light information region. However, this result can be achieved in any way (e.g., by processing the information cold light region and the non-information cold light region separately with the same or different operations, by processing only the information cold light region while leaving the non-information cold light region unchanged, darkened, or canceled, etc.).

[0081] In this embodiment, the processing of the cold light image is based on the cold light values ​​of the cold light information region to facilitate the identification of the target representation therein. However, this processing can be based on these cold light values ​​in any way (e.g., based on their distribution, range, probability, etc.) and in any way (e.g., by making the target representation more significant for its manual identification, by automatically identifying the target representation, etc.).

[0082] In an embodiment, the method includes (by a computing device) outputting an output image based on a processed cold light image. However, the output image can be of any type (e.g., the same processed cold light image, a processed cold light image combined with / overlapped with a reflected image, etc.), and it can be output in any manner (e.g., displayed, printed, transmitted remotely, in real time or offline, etc.).

[0083] Other embodiments provide additional advantageous features; however, these features may be omitted entirely in the basic implementation.

[0084] Specifically, in this embodiment, the method is used to assist in surgical procedures for a patient. However, the surgical procedure can be of any type (e.g., for therapeutic purposes, preventative purposes, aesthetic purposes in standard surgery, minimally invasive procedures such as laparoscopy, arthroscopy, angioplasty, etc.).

[0085] In an embodiment, the step of outputting the image includes (by a computing device) and displaying the output image substantially in real time using the step of acquiring the cold light image. However, the output image can be displayed in any manner (e.g., on any display unit such as a monitor, virtual reality glasses, etc., with any delays caused by its generation, etc.).

[0086] In this embodiment, the area of ​​concern is the patient's surgical cavity. However, the surgical cavity can be of any type (e.g., a wound, an open body, etc.).

[0087] In one embodiment, at least a portion of the foreign object overlaps with the region of interest. However, the foreign object can be positioned in any location (e.g., overlapping with the region of interest to any extent, bypassing it, being separated from it, or any combination thereof).

[0088] In embodiments, foreign bodies include one or more medical devices, one or more hands, one or more medical tools, one or more body parts of the patient that are not of medical procedure concern, and / or one or more background materials. However, foreign bodies can include any number and type of medical devices (e.g., surgical instruments such as scalpels, scissors, endoscopic instruments such as manipulators, samplers, polyp removal traps, etc.), hands (e.g., the hands of surgeons, assistants, nurses, etc.), medical tools (such as surgical instruments such as gauze, traction devices, curtains, coverings, endoscopic tools such as hemostatic clips, irrigators, etc.), body parts of non-concern (e.g., body parts surrounding the area of ​​concern, such as skin and muscles around the surgical cavity, organs unrelated to the medical procedure, such as the liver, etc.), background materials (e.g., operating tables, walls, floors, etc.), or more generally localized, different, and additional background materials (e.g., biological materials of the patient that disrupt the medical procedure, such as fecal residue during colonoscopy, food residue during gastroscopy, etc.).

[0089] In this embodiment, the auxiliary image is a reflected image, the auxiliary light is visible light, and the auxiliary value represents the visible light reflected from the corresponding auxiliary position of the field of view illuminated by white light. However, the white light (and the corresponding visible light) can be of any type (e.g., any non-cold light that does not exhibit significant cold light phenomena for cold light materials).

[0090] In this embodiment, the photoluminescent substance is a photoluminescent agent pre-administered to the patient prior to the execution of the method. However, the photoluminescent agent can be of any type (e.g., any targeted photoluminescent agent, such as those based on specific or non-specific interactions, any non-targeted photoluminescent agent, etc.), and it can be pre-administered in any manner (e.g., using a syringe, infusion pump, etc.) and at any time (e.g., in advance, before the execution of the method, continuously during the execution of the method, etc.). In any case, this is a data processing method that can be implemented independently of any interaction with the patient; furthermore, the photoluminescent agent can be administered to the patient non-invasively (e.g., orally to image the gastrointestinal tract, via a nebulizer into the respiratory tract, via local spray application, or local introduction during a medical procedure, etc.), or in any case without any substantial physical intervention on the patient that would require specialized medical expertise or pose any health risk to him / her (e.g., intramuscular).

[0091] In one embodiment, the step of identifying the auxiliary information region includes semantically segmenting the auxiliary image (by a computing device). However, the auxiliary image can be semantically segmented in any way (e.g., using classification algorithms, neural networks, etc.).

[0092] In an embodiment, the auxiliary image is semantically segmented into auxiliary information regions corresponding to at least one category of the region of interest and auxiliary non-information regions corresponding to one or more categories of foreign objects. However, the categories of the region of interest and the categories of foreign objects can have any number and any type (e.g., a single category of the region of interest for the entire region of interest, a single category of foreign objects for all foreign objects, multiple categories of the region of interest for a corresponding portion or group thereof, multiple categories of foreign objects for a corresponding type of foreign object or group thereof, etc.).

[0093] In an embodiment, the step of segmenting the auxiliary image includes (by a computing device) semantically segmenting the auxiliary image using a neural network. However, the neural network can be of any type (e.g., U-network, convolutional neural network, feedforward neural network, radial basis function neural network, recurrent neural network, modular neural network, etc.). The neural network can be trained in any manner (e.g., based on stochastic gradient descent, real-time recurrent learning, higher-order gradient descent techniques, extended Kalman filtering, and similar algorithms) with any number and type of training pairs (e.g., randomly, homogeneously selected, etc.).

[0094] In an embodiment, the step of segmenting the auxiliary image includes (by a computing device) determining one or more feature maps corresponding to features of the auxiliary image. However, the features can be of any number and any type (e.g., selected using any heuristic, iterative, filtering, or other methods relative to partial, distinct, and additional candidate features mentioned above).

[0095] In this embodiment, each feature map includes a corresponding feature value for an auxiliary location. However, the feature values ​​can be determined in any way (e.g., using any filter, neural network, encoder, corrector, etc.).

[0096] In an embodiment, the method includes (by a computing device) semantically segmenting an auxiliary image by applying a classification algorithm to feature values ​​of a feature map. However, the classification algorithm can be of any type (e.g., Conditional Random Field, Markov Random Field, SIOX, GrabCut, Decision Tree, k-Nearest Neighbors, etc.). The classification algorithm can be configured in any way (e.g., utilizing any heuristic, iterative, filtering, etc. methods for any of its parameters).

[0097] In an embodiment, the step of identifying the auxiliary information region includes, when the auxiliary image includes one or more disconnected portions completely surrounded by the auxiliary information region, assigning the disconnected portions to the auxiliary information region (by a computing device). However, this hole-filling step can be performed in any manner (e.g., indiscriminately or only for disconnected portions greater than a threshold, whether for classification algorithms or deep learning techniques, to auxiliary images or cold light images, etc.).

[0098] In an embodiment, the method includes preprocessing the auxiliary image (by a computing device) before the identification of the auxiliary information region. However, the auxiliary image may undergo any number and type of preprocessing steps (relative to the partial, different, and additional steps above), or even none at all.

[0099] In an embodiment, the method includes (by a computing device) preprocessing the auxiliary image by applying histogram equalization to the auxiliary image before segmenting it. However, the histogram equalization can be performed in any manner (e.g., using ordinary histogram equalization, adaptive histogram equalization, contrast-finite adaptive equalization, and similar algorithms).

[0100] In an embodiment, histogram equalization is performed in response to a luminance indicator of an auxiliary image included between a darkness threshold (indicating the feasibility of the recognition assistance information region) and a luminance threshold (above the darkness threshold). However, the luminance indicator can be calculated in any way (e.g., as an average, modulus, median, etc.), and the darkness / luminance thresholds can have arbitrary values ​​(e.g., predefined, dynamically determined, etc.); in any case, the possibility of performing histogram equalization indiscriminately (always) or never is not excluded.

[0101] In an embodiment, the step of identifying the auxiliary information region includes (by a computing device) further identifying the auxiliary information region based on the cold light image. However, the cold light image can be used to identify the auxiliary information region in any way (e.g., by also inputting the cold light image into a neural network, by extracting one or more additional feature maps from the cold light image for use by a classification algorithm, directly or weighted to limit the influence of the cold light image, etc.).

[0102] In one embodiment, the step of generating the processed cold light image includes (by a computing device) automatically scaling the cold light information region based on the cold light value of the cold light information region. However, the cold light information region can be automatically scaled in any way (e.g., mapping its cold light value, logarithmic compression, saturation, etc.).

[0103] In an embodiment, the step of automatically scaling the cold light information region includes (by a computing device) determining the cold light range of the cold light values ​​of the cold light information region. However, the cold light range can be determined in any way (e.g., indiscriminately, by ignoring outliers, etc.).

[0104] In an embodiment, the step of automatically scaling the cold light information region includes (by a computing device) converting the cold light value of the cold light information region according to a mapping function that maps the cold light range to a display range for displaying the cold light image. However, the mapping function can be of any type (e.g., nonlinear, such as logarithmic, exponential, linear, etc.).

[0105] In an embodiment, the step of generating the processed cold light image includes (by a computing device) thresholding the cold light information region based on the cold light value of the cold light information region, thereby dividing the cold light information region into target segments representing a target body and non-target segments representing the remaining portion that is different from the target body's region of interest. However, the cold light information region can be thresholded in any way (e.g., based on binary, multi-level or multi-band statistical distributions, entropy, clustering, or object attributes, etc.).

[0106] In an embodiment, the step of outputting the image includes (by a computing device) outputting the image by highlighting the target segment relative to the non-target segment. However, the target segment can be highlighted in any way (e.g., by masking the non-target segment, by representing the target segment in color and the non-target segment in black and white, by increasing and / or decreasing the brightness of the target segment and the non-target segment respectively, etc.).

[0107] In this embodiment, the step of thresholding the cold light information region includes (by a computing device) determining a threshold based on the statistical distribution of the cold light values ​​of the cold light information region. However, the threshold can be determined in any way (e.g., processing bimodal, unimodal, multimodal, or other statistical distributions).

[0108] In an embodiment, the step of thresholding the cold light information region includes (by a computing device) assigning each cold light position of the cold light information region to a target segment or a non-target segment based on a comparison of the corresponding cold light value with a threshold. However, the cold light positions can be assigned to the target / non-target segments according to the threshold in any way (e.g., by generating a thresholded cold light image or a thresholded mask, etc., when they are high and / or low).

[0109] In an embodiment, the step of thresholding the cold light information region includes (by a computing device) calculating one or more target region statistical parameters for the cold light value of the target region segment and / or one or more non-target region statistical parameters for the non-target region segment. However, the target / non-target region statistical parameters can have any number, each of which can be zero, and can be of any type, whether the same or different (e.g., mean, median, modulus, standard deviation, variance, skewness, etc.).

[0110] In an embodiment, the step of generating the processed cold light image includes updating the cold light value of the target segment (by a computing device) according to target region statistical parameters and / or non-target region statistical parameters, respectively. However, the target segment can be processed in any way according to these statistical parameters (e.g., according to target region statistical parameters only, according to non-target region statistical parameters only, according to both, partial, different, and additional processing relative to the above processing, whether independently or in any combination thereof, etc.).

[0111] In an embodiment, the method includes (by a computing device) determining, based on its content, a corresponding optical value (of at least one optical parameter related to cold light) for an auxiliary position of an auxiliary image limited to an auxiliary information region. However, the optical parameters can have any number and any type (e.g., partial, different, and additional optical parameters relative to the aforementioned optical parameters, etc.), and the corresponding optical values ​​can be determined in any way (e.g., independently based on the corresponding auxiliary value, by applying any classification algorithm, etc.).

[0112] In an embodiment, the step of generating the processed cold light image includes (by a computing device) equalizing the cold light values ​​of the cold light information region based on optical values. However, the cold light information region can be equalized at any time (e.g., before and / or after autoscaling / thresholding, etc.) in any manner (e.g., by equalizing only the optical values ​​of the corresponding auxiliary position, and further by equalizing the optical values ​​of its adjacent auxiliary positions, relative to the portion, difference, and additional further processing described above, whether independently or in any combination thereof, etc.).

[0113] In this embodiment, the cold-light material is a fluorescent material (where the cold-light image is a fluorescent image, and the cold-light value represents the fluorescence emitted by the fluorescent material at the corresponding cold-light location illuminated by its excitation light). However, the fluorescent material can be of any extrinsic / intrinsic or exogenous / endogenous type (e.g., for imaging any pathological tissue, any healthy tissue, etc.).

[0114] Generally speaking, similar considerations apply if the same solution is achieved using equivalent methods (by using similar steps with more steps or parts of the same functionality, removing some unnecessary steps, or adding additional optional steps); furthermore, these steps can be performed in different orders, concurrently, or in an interleaved manner (at least partially).

[0115] An embodiment provides a computer program configured to cause the computing device to perform the methods described above when executed on a computing device. An embodiment provides a computer program product comprising one or more computer-readable storage media having program instructions commonly stored on the storage media, which can be loaded by a computing device to cause the computing device to perform the same methods. However, the computer program can be implemented as a standalone module, a plug-in for a pre-existing software program (e.g., a manager for an imaging system), or even directly in the latter. Similar considerations apply if the computer program is constructed differently, or if additional modules or functions are provided; similarly, the memory structure can have other types, or can be replaced by an equivalent entity (not necessarily including physical storage media). The computer program can take any form suitable for use by any computing device (see below) to configure the computing device to perform the desired operation; specifically, the computer program can be in the form of external or resident software, firmware, or microcode (in the form of object code or source code, e.g., compiled or interpreted). Furthermore, the computer program can be provided on any computer-readable storage medium. Storage media are any tangible medium (unlike transient signals themselves) capable of holding and storing instructions for use by a computing device. For example, storage media can be electrical, magnetic, optical, electromagnetic, infrared, or semiconductor; examples of such storage media are fixed disks (where programs can be pre-loaded), removable disks, memory keys (e.g., USB type), etc. Computer programs can be downloaded to the computing device from storage media or via a network (e.g., the Internet, wide area network, and / or local area network, including transmission cables, fiber optics, wireless connections, network devices); one or more network adapters in the computing device receive the computer program from the network and forward it to one or more storage devices in the computing device for storage. In any case, the solutions according to embodiments of this disclosure are implemented even using hardware structures (e.g., electronic circuitry integrated into one or more chips of semiconductor material, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits), or using a combination of appropriately programmed or otherwise configured software and hardware.

[0116] An embodiment provides a computing device including components configured to perform the steps of the method described above. An embodiment also provides a computing device including circuitry (i.e., any hardware appropriately configured by software) for performing each step of the same method. However, the computing device can be of any type (e.g., a central unit of an imaging system, a separate computer, etc.).

[0117] The embodiments provide an imaging system. However, the imaging system can be of any type (e.g., a guided surgical device, an endoscope, a laparoscope, etc.).

[0118] In this embodiment, the imaging system includes the computing device described above. However, the computing device can be provided in the imaging system in any way (e.g., embedded, connected via any wired / wireless connection, etc.).

[0119] In one embodiment, the imaging system includes an illumination unit for applying excitation light suitable for exciting a cold light substance to the field of view. However, the illumination unit can be of any type (e.g., laser-based, LED, UV / halogen / xenon lamp, whether white light is provided, etc.).

[0120] In this embodiment, the imaging system includes an acquisition unit for acquiring cold light images and auxiliary images. However, the acquisition unit can be of any type (e.g., based on any number and type of lenses, light guides, mirrors, CCDs, ICCDs, EMCCDs, CMOS, InGaAs, or PMT sensors, etc.).

[0121] Generally, similar considerations apply if the computing device and imaging system each have different structures or include equivalent components, or if it has other operational characteristics. In any case, each component can be divided into multiple elements, or two or more components can be combined into a single element; furthermore, each component can be replicated to support the parallel execution of corresponding operations. Moreover, unless otherwise stated, any interaction between different components generally need not be sequential, and it can be direct or indirect through one or more intermediaries.

[0122] An embodiment provides a medical procedure for a patient, comprising the following steps: Imaging a field of view containing a region of interest of the medical procedure for the patient, according to the method described above, to output an output image; the region of interest includes at least one target of the medical procedure containing a photoluminescent material and one or more foreign objects distinct from the region of interest. The medical procedure is performed with the aid of the output image. However, the proposed method can be applied to any type of medical procedure (see above).

[0123] In one embodiment, the medical procedure includes applying a light-emitting agent containing a light-emitting substance to the patient. However, the light-emitting agent can be applied in any manner (see above), or this step can be omitted entirely (if the light-emitting agent is endogenous).

Claims

1. A method for imaging a region of interest of a patient undergoing a medical procedure, wherein the method includes, under the control of a computing device: A cold light image of a field of view is acquired by a computing device. The field of view includes the region of interest and one or more foreign objects distinct from the region of interest. The region of interest includes at least one target containing a cold light material in the medical procedure. The cold light image includes multiple cold light values ​​representing the cold light emitted by the cold light material at corresponding cold light locations in the field of view. An auxiliary image of the field of view is acquired by a computing device. The auxiliary image includes multiple auxiliary values ​​representing auxiliary light received from corresponding auxiliary positions in the field of view that is different from the cold light. The computing device semantically segments the auxiliary values ​​of the auxiliary image into at least one region of interest category and one or more foreign object categories. The at least one region of interest category defines an auxiliary information region representing a region of interest without foreign objects, and the one or more foreign object categories define a non-information region representing a foreign object. The computing device identifies the cold light value of the cold light image, which corresponds to the auxiliary value of the auxiliary information region of the auxiliary image, defining the cold light information region. A processed cold light image is generated by a computing device by processing the cold light image limited to the cold light information region. The processed cold light image includes multiple processed cold light values ​​based on the cold light values ​​of the cold light information region. The processed cold light image provides a further representation of the target to facilitate target identification. The computing device displays an output image based on the acquired cold light image in essentially real time.

2. The method according to claim 1, wherein, At least a portion of the foreign object overlaps with the region of interest.

3. The method according to claim 1 or 2, wherein, The foreign object includes one or more medical devices, one or more hands, one or more body parts of the patient that are not of concern in the medical procedure, and / or background material.

4. The method according to claim 1 or 2, wherein, The auxiliary image is a reflected image, the auxiliary light is visible light, and the auxiliary value represents the visible light reflected from the corresponding auxiliary position of the field of view illuminated by white light.

5. The method according to claim 1 or 2, wherein, The photoluminescent substance is a photoluminescent agent that is pre-applied to the patient before performing the method.

6. The method according to claim 1, wherein, The auxiliary values ​​of the segmentation auxiliary image include: The auxiliary values ​​of the auxiliary image are semantically segmented by a computing device using a neural network.

7. The method according to claim 1, wherein, The auxiliary values ​​of the segmentation auxiliary image include: A computing device determines one or more feature maps corresponding to the auxiliary values ​​of the auxiliary image, each feature map including the corresponding feature value of the auxiliary location, and The auxiliary values ​​of the auxiliary image are semantically segmented by a computing device by applying a classification algorithm to the feature values ​​of the feature map.

8. The method according to claim 1 or 2, wherein, The auxiliary values ​​for segmenting the auxiliary image include those when the auxiliary image includes one or more disconnected portions completely surrounded by the auxiliary information region: The computing device assigns the disconnected portion to the auxiliary information region.

9. The method according to claim 1 or 2, wherein, The method includes: In response to the brightness indicator of the auxiliary image being included between a darkness threshold indicating the feasibility of segmenting the auxiliary value of the auxiliary image and a brightness threshold above the darkness threshold, the auxiliary image is preprocessed by the computing device by applying histogram equalization to the auxiliary value of the auxiliary image before segmenting the auxiliary value of the auxiliary image.

10. The method according to claim 1 or 2, wherein, The auxiliary values ​​for segmenting the auxiliary image include: The auxiliary information region is further identified by the computing device based on the cold light value of the cold light image.

11. The method according to claim 1 or 2, wherein, The generation of the processed cold light image includes: The computing device automatically scales the cold light information region based on the cold light value of the cold light information region.

12. The method according to claim 11, wherein, The automatically scaled cold light information area includes: The cold light range of the cold light value of the cold light information region is determined by a computing device, and The computing device converts the cold light value of the cold light information area according to a mapping function that maps the cold light range to the display range used to display the cold light image.

13. The method according to claim 1 or 2, wherein, The generation of the processed cold light image includes: The computing device thresholds the cold light information region based on its cold light value, thereby dividing the cold light information region into a target segment representing the target body and a non-target segment representing the region of interest, which is different from the target body. And the displayed output image includes: The output image is displayed by a computing device by highlighting the target region relative to the non-target region.

14. The method according to claim 13, wherein, The thresholding of the cold light information region includes: The threshold is determined by a computing device based on the statistical distribution of the cold light values ​​in the cold light information region, and The computing device assigns each cold light position of the cold light information region to the target segment or the non-target segment based on a comparison between the corresponding cold light value and the threshold.

15. The method according to claim 13, wherein, The thresholding of the cold light information region includes: One or more target region statistical parameters for calculating the cold light value of the target region segment and / or one or more non-target region statistical parameters for calculating the cold light value of the non-target region segment by a computing device, and The computing device updates the cold light values ​​of each segment of the target region based on the target region statistical parameters and / or the non-target region statistical parameters.

16. The method according to claim 1 or 2, wherein, The method includes: The computing device, based on the content of the auxiliary image (limited to the auxiliary information region) at the auxiliary position, determines the corresponding optical value of at least one optical parameter related to the cold light. And the generation of the processed cold light image includes: The computing device equalizes the cold light value of the cold light information region based on the optical value.

17. The method according to claim 1 or 2, wherein, The cold light material is a fluorescent material, the cold light image is a fluorescent image, and the cold light value represents the fluorescence emitted by the fluorescent material at the corresponding cold light position illuminated by its excitation light.

18. A computer-readable storage medium storing program instructions configured to cause a computing device to perform the method according to any one of claims 1 to 17.

19. A computer program product comprising program instructions that can be read by a computing device to cause the computing device to perform the method according to any one of claims 1 to 17.

20. A computing device comprising means configured to perform the steps of the method according to any one of claims 1 to 17.

21. An imaging system, comprising: The computing device according to claim 20, An illumination unit for applying excitation light suitable for exciting a cold light substance to the field of view, and The acquisition unit is used to acquire cold light images and auxiliary images.

Images