Support for medical treatment using a luminescence image processed in a restriction information area identified in a corresponding auxiliary image

Abstract

A technique for assisting a medical procedure is proposed. A corresponding method (400) includes acquiring (406-408) a luminescence image (205F) based on luminescent light of a field of view (103) and acquiring (406-408) an auxiliary image (205R) based on auxiliary light different from the luminescent light. The field of view (103) contains a region of interest (109) including a target body (112) of the medical procedure (containing a luminescent material) and one or more foreign objects (115-124). Auxiliary information regions (210Ri) representing the region of interest (109) without the foreign objects (115-124) are identified (410-424) in the auxiliary image (205R) according to their content, and luminescence information regions (210Fi) are identified (428) in the luminescence image (205F) according to the auxiliary information regions (210Ri). The luminescent image (205F) is processed (432-466) to limit the luminescent information region (210Fi) to facilitate identification of the representation of the target body (112) therein.

Description

Technical Field

[0001] The present disclosure relates to imaging applications. More particularly, the present disclosure relates to luminescence imaging for assisting medical treatment.

Background Art

[0002] The background of the present disclosure is introduced below together with a discussion of the technology relevant to its context. However, even if this discussion refers to documents, acts, artifacts, etc., it is not intended to imply or represent that the technology discussed is part of the prior art or common general knowledge in the field related to the present disclosure.

[0003] Luminescence imaging, particularly fluorescence imaging, is a particular imaging technique used to acquire images that provide a visual representation of an object even when they are not directly visible. Luminescence imaging is based on the luminescence phenomenon consisting of the emission of light by a luminescent substance when it is excited by an excitation different from heating, and in particular, the fluorescence phenomenon occurs in a fluorescent substance (referred to as a fluorescent dye molecule). These emit light (fluorescence) when illuminated.

[0004] Imaging techniques are commonly used in medical devices to examine a patient's body part (internally) to assist in medical treatment. For example, in fluorescence-guided surgery (FGS) (also called fluorescence-guided resection (FGR) when related to tumors), a fluorescent agent (possibly adapted to reach specific molecules of a desired target body such as a tumor and become immobilized there) is administered to the patient. Visualization of the fluorescent agent in the corresponding fluorescence image generally overlaps on the corresponding reflection image, facilitating a surgeon's operation, for example, the recognition of a tumor to be resected.

[0005] However, the imaged field of view often includes several foreign objects of no interest in addition to the actual region of interest. For example, in (fluorescent) guided surgery, this can be due to the presence of surgical instruments, hands, surgical tools, surrounding body parts (e.g., skin around the surgical cavity or irrelevant organs within it), and background material in addition to the surgical cavity. Foreign objects generate spurious light in addition to the actually interesting fluorescent light emitted by the fluorescent agent accumulated in the target body of the medical treatment (the tumor in the example under discussion). In particular, spurious light can increase the fluorescent light. For example, this can be due to scattering and absorption phenomena. Furthermore, this can also be due to the fluorescent agent accumulating in surrounding body parts (especially the skin) due to their (undesirable) affinity. Conversely, spurious light can abnormally or artificially decrease the fluorescence.

[0006] Spurious light (which is completely non-informative) is harmful to the imaging of the tumor (or any other target body). In particular, spurious light significantly biases the statistical distribution of the (fluorescent) values of the fluorescent image. This has an adverse effect on the subsequent processing of the fluorescent image. For example, the fluorescent values are generally converted from the (fluorescent) range given by the measured values of the fluorescent light to the (display) range given by the display dynamics of the monitor used to display the fluorescent image.

[0007] Therefore, the bias of the statistical distribution of the fluorescent values by spurious light (when increasing the fluorescence) limits the range of fluorescent values used to display the region of interest. This makes the representation of the tumor less prominent. Furthermore, fluorescent images are often thresholded to distinguish the tumor from the rest of the fluorescent image according to the comparison of its fluorescent values with a threshold. The threshold is automatically calculated according to the fluorescent values.

[0008] Also in this case, the bias in the statistical distribution of the fluorescence values due to spurious light (whether increasing or decreasing the fluorescence) affects the threshold value. This entails the risk of misclassifying the fluorescence values. As a result, there is a possibility of over-detection of tumors (false positives), in particular under-detection of tumors (false negatives). Furthermore, this inhibits the detection of tumor lesions. All of the above may have serious consequences for the patient's health (e.g., incomplete resection of the tumor).

[0009] The involvement of spurious light is difficult (if not impossible) to remove from the fluorescence light of interest. In fact, since spurious light cannot be easily distinguished statistically, statistical methods are not at all effective for this purpose. In particular, when the spurious light has the same spectral characteristics as the fluorescence light of interest (e.g., in the case of a fluorescent agent accumulated in the skin), optical filtering is also not at all effective. Furthermore, manual adjustment of the operating parameters of the medical device (e.g., including covering the foreign object with a non-fluorescent material) to limit the influence of spurious light adds further work (possibly requiring a dedicated operator who needs specific training). In any case, the interaction with the medical device may become difficult during the surgical procedure, especially due to concerns about sterilization.

[0010] International Publication No. WO-A-2013 / 096766 discloses a method for imaging lesions in diagnostic applications. The mole boundary is placed within the visible light image. The visible light image and the fluorescence image are aligned using positioning references for both of them. Mole features are extracted from one or both images.

[0011] International Publication No. WO-A-2019 / 232473 discloses a method for automatically detecting and characterizing micro-objects, such as cells or beads, placed within a microfluidic device. The pixel data in the illuminated image is processed using a neural network to detect the micro-objects. Signals located within the corresponding boundaries of each detected micro-object in a non-irradiated image, such as a fluorescence image, are used to measure the characteristics of the micro-objects.

[0012] International Publication No. WO-A-2017 / 098010 discloses a method for distinguishing live beads from blank beads in DNA / RNA sequencing. The position of the beads is determined in a white light illumination image. The beads at the positions thus determined are classified according to the emission of electromagnetic radiation by a fluorescent compound. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0013] The simplified gist of the present disclosure is presented here to provide a basic understanding thereof. However, the sole purpose of this gist is to introduce some concepts of the present disclosure in a simplified form as a prelude to the more detailed description below. It should not be construed as an identification of its key elements nor as a delineation of its scope.

[0014] Generally speaking, the present disclosure is based on the idea of processing a luminescence image restricted to an information region identified in an auxiliary image.

[0015] In particular, one aspect provides a method for assisting medical treatment. This method includes the step of acquiring a luminescence image (based on luminescence light) and an auxiliary image (based on auxiliary light different from this luminescence light) of a visual field. The visual field accommodates a region of interest including a target body (including a luminescent substance) for medical treatment and one or more foreign substances. An auxiliary information region representing the region of interest without foreign substances is identified in the auxiliary image according to its content, and a luminescence information region is identified in the luminescence image according to the auxiliary information region. The luminescence image is processed limited to the luminescence information region to facilitate the identification of the representation of the target body therein.

[0016] A further aspect provides a computer program for implementing this method.

[0017] A further aspect provides a corresponding computer program product.

[0018] A further aspect provides a computer device for performing this method.

[0019] A further aspect provides an imaging system including this computer device.

[0020] A further aspect provides a corresponding medical treatment.

[0021] More particularly, one or more aspects of the present disclosure are described in the independent claims, and the advantageous features thereof are described in the dependent claims, and the wording of all the claims is incorporated herein by reference in its entirety (with any advantageous features provided with reference to any specific aspect applicable mutatis mutandis to all other aspects).

Brief Description of the Drawings

[0022] The methods of the present disclosure, and its additional features and advantages, will be best understood with reference to the following detailed description, taken in conjunction with the accompanying drawings, which are presented by way of non-limiting illustration only (wherein, for the sake of simplicity, corresponding elements are provided with equal or similar reference numerals and their description is not repeated, and the name of each entity is generally used to indicate both its type and its attributes (e.g., value, content, and representation)).

[0023]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 2E

Figure 3

Figure 4A

Figure 4B

Figure 5

DETAILED DESCRIPTION OF THE INVENTION

[0024] Referring particularly to FIG. 1, a schematic block diagram of an imaging system 100 that can be used to practice the method according to an embodiment of the present disclosure is shown.

[0025] The imaging system 100 is capable of imaging a corresponding field of view 103 (defined by a portion of the world within the solid angle to which the imaging system 100 is sensitive). In particular, the imaging system 100 is used in surgical applications (FGS, particularly FGR) to assist a surgeon. In this particular case, the field of view 103 is related to a patient 106 undergoing a surgical procedure to which a fluorescent agent has been administered (e.g., adapted to accumulate in a tumor). The field of view 103 includes a surgical cavity 109 (e.g., a small skin incision in minimally invasive surgery). This is open within the patient 106 and exposes the corresponding body part undergoing the surgical procedure.

[0026] In particular, the body part exposed within the surgical cavity 109 includes the target body and what the surgeon should operate on it, for example, the tumor 112 to be resected. The field of view 103 generally includes one or more foreign objects (different from the surgical cavity 109). For example, these foreign objects may include one or more surgical instruments 115 (e.g., surgical knives), one or more hands 118 (e.g., the surgeon's hand), one or more surgical tools 121 (e.g., gauze), one or more surrounding body parts 124 (e.g., the skin near the surgical cavity 109), and / or one or more background materials 125 (e.g., the operating table).

[0027] The imaging system 100 has an imaging probe 127 for acquiring an image of the field of view 103 and a central unit 130 for controlling its operation.

[0028] Starting from the imaging probe 127, it has an illumination unit (for illuminating the field of view 103) and an acquisition unit (for acquiring an image of the field of view 103) comprising the following components. In the illumination unit, an excitation light source 133 and a white light source 136 generate excitation light and white light respectively. The excitation light has a wavelength and energy (e.g., near-infrared (or NIR) type) suitable for exciting the fluorescent dye molecules of the fluorescent agent, while the white light appears to be substantially colorless to the human eye (e.g., includes all wavelengths in the spectrum visible to the human eye with equal intensity).

[0029] The corresponding transmission optical systems 139 and 142 transmit the excitation light and the white light to the same field of view 103, respectively. In the acquisition unit, the collection optical system 145 collects the light from the field of view 103 (in an epi-illumination arrangement). The collected light includes fluorescence emitted by any fluorescent dye molecules present within the field of view (illuminated by the excitation light). In fact, when a fluorescent dye molecule absorbs the excitation light, it enters an excited (electronic) state, and the excited state is unstable. As a result, the fluorescent dye molecule decays from there to the ground (electronic) state in an extremely short time, thereby emitting fluorescent light at an intensity that depends on the amount of the illuminated fluorescent dye molecules (at a characteristic wavelength longer than that of the excitation light due to the energy dissipated as heat in the excited state), and other factors include the positions of the fluorescent dye molecules within the field of view and the body part.

[0030] Furthermore, the collected light includes reflected light (in the visible spectrum) reflected by any object present within the field of view (illuminated by the white light). The beam splitter 148 divides the collected light into two channels. For example, the beam splitter 148 is a dichroic mirror that transmits and reflects the collected light at wavelengths above and below a threshold wavelength between the spectrum of the reflected light and the spectrum of the fluorescent light (or vice versa). In the (transmitted) channel of the beam splitter 148, using the fluorescent light defined by a portion of the collected light in its spectrum, the emission filter 151 filters the fluorescent light to remove the excitation / white light (which may be reflected by the field of view) and the ambient light (which may be generated by intrinsic fluorescence).

[0031] The fluorescence camera 154 (e.g., EMCCD type) receives the fluorescent light from the emission filter 151 and generates a corresponding fluorescent (digital) image representing the distribution of the fluorescent dye molecules within the field of view 103. On the other side of the other (reflected) channel of the beam splitter 148, it has the reflected light defined by a portion of the collected light in its spectrum, and the reflectance or photographic camera 157 receives the reflected light and generates a corresponding reflected (digital) image representing what is visible within the field of view 103.

[0032] When moving to the central unit 130, it comprises several units connected to each other via a bus structure 160. In particular, one or more microprocessors (μP) 163 provide the logical capabilities of the central unit 130. A non-volatile memory (ROM) 166 stores the basic code for the bootstrap of the central unit 130, and a volatile memory (RAM) 169 is used as a working memory by the microprocessor 163. The central unit 130 is provided with a large-capacity memory 172 (e.g., a semiconductor disk or SSD) for storing programs and data.

[0033] Furthermore, the central unit 130 comprises a plurality of controllers 175 for peripheral devices or input / output (I / O) units. In particular, the controller 175 controls the excitation light source 133, the white light source 136, the fluorescence camera 154 and the reflection camera 157 of the imaging probe 127. Further, the controller 175 controls further peripheral devices (shown generally by reference numeral 178), for example, one or more monitors for image display, a keyboard for command input, a trackball for moving a pointer on the monitor, a drive for reading and writing a removable storage unit (e.g., a USB key), and a network interface card (NIC) for connecting to a communication network (e.g., a LAN).

[0034] Referring now to FIGS. 2A - 2E, various application examples of the method according to an embodiment of the present disclosure are shown.

[0035] Starting from FIG. 2A, a pair of corresponding reflected image 205R and fluorescence image 205F are shown. The reflected image 205R and the fluorescence image 205F provide a simultaneous representation (from the viewpoints of reflected light and fluorescence light respectively) of the same field of view 103. In particular, the field of view 103 contains the surgical cavity 109 and several foreign objects, which in this example consist of the surgical instrument 115, the hands 118 of two surgeons, and the surrounding skin 124. The foreign objects 115 - 124 are arranged around the surgical cavity 109 (such as in the case of the surgical instrument 115, a part of the surgeon's hands 118 and the surrounding skin 124), or may overlap with the surgical cavity 109 (such as in the case of a part of the surgeon's hands 118).

[0036] Referring to FIG. 2B, in the method according to an embodiment of the present disclosure, in the reflected image 205R, an (reflected) information region 210Ri is identified according to its content. The information region 210Ri represents the surgical cavity without foreign objects (in this case, the surgical instrument, the surgeon's hands, the surrounding skin), and represents the information part (i.e., the region of interest or ROI) of the actually concerned field of view 103. The rest of the reflected image 205R defines a (reflectivity) non - information region 210Rn representing foreign objects and defines the non - information part of the field of view 103 that is not of interest. For example, as will be described in detail below, this result is achieved using a semantic segmentation technique (e.g., based on the use of a neural network).

[0037] The identification of the information region 210Ri (similarly for the non - information region 210Rn) in the reflected image 205R is transferred to the fluorescence image 205F. In particular, in the fluorescence image 205F corresponding to the information region 210Ri, a (fluorescence) information region 210Fi is identified. As a result, the rest of the fluorescence image 205F defines a (fluorescence) non - information region 210Fn corresponding to the non - information region 210Rn.

[0038] As will be described in detail below, a processed image is generated by processing the fluorescence image 205F limited to the information region 210Fi. The processing of the fluorescence image 205F is based on, for example, the (fluorescence) value of the fluorescence image 205F only in the information region, for example, its distribution (e.g., range, probability, etc.). This processing of the fluorescence image 205F aims to facilitate the identification of the expression of the tumor therein (e.g., by auto-scaling (automatically enlarging and reducing) or threshold processing the information region 210Fi).

[0039] As a result, it is possible to consider only the (information) expression of the surgical cavity in the fluorescence image 205F, and instead, the (non-information) expression of foreign substances (surrounding and / or overlapping) can be ignored. This avoids (or at least substantially reduces) the adverse effects of foreign substances in the imaging of the surgical cavity. In particular, the statistical distribution of the fluorescence values based on the processing of the fluorescence image 205F is unbiased (because the fluorescence values in the non-information region 210Fn do not contribute to it).

[0040] For example, FIG. 2C shows curves 215w and 215i that represent the entire fluorescence image and the corresponding probability functions of the fluorescence values of only its information region, respectively. The probability functions 215w, 215i approximate the corresponding histograms of the fluorescence values in a qualitative diagram plotting the fluorescence values on the horizontal axis and the frequency on the vertical axis. As can be seen from the figure, the probability function 215i is much narrower than the probability function 215w. Therefore, the processing of the fluorescence image limited to the information region benefits from a narrower probability distribution of its fluorescence values.

[0041] In particular, FIG. 2D shows a processed image generated by auto-scaling the entire fluorescence image, called the auto-scale (fluorescence) image 220Fw. The auto-scale image 220Fw is obtained by applying a mapping function to all the fluorescence values of the fluorescence image in order to convert them from the (fluorescence) range given by the measurement resolution of the fluorescence light to the (display) range given by the display dynamics of the monitor used to display it (e.g., based on the logarithmic law, an image with a balanced contrast is obtained).

[0042] The figure also shows a processed image generated by auto - scaling the fluorescence image limited to the information region 210Fi. (And by further auto - scaling the fluorescence image limited to the non - information region 210Fn), it is called the auto - scaled (fluorescence) image 220Fi. In particular, the auto - scale image 220Fi is obtained by separately applying corresponding mapping functions to the fluorescence values in the information region 210Fi and the fluorescence values in the non - information region 210Fn (the fluorescence values in the non - information region 210Fn are reduced by a scaling factor for masking its content).

[0043] In the auto - scale image 220Fw, its wide statistical distribution of fluorescence values restricts the range of available fluorescence values within the display range for mapping the fluorescence values in the information region 210Fi (due to their narrower statistical distributions). This reduces the difference in fluorescence values in the information region 210Fi, thereby making it extremely difficult (if not impossible) to identify that part at a higher concentration of the fluorescent agent representing the tumor to be resected.

[0044] In the auto - scale image 220Fi, instead, the entire display range is available for mapping the fluorescence values in the information region 210Fi. This increases the difference in fluorescence values in the information region 210Fi, making the representation of the tumor 112 more prominent.

[0045] Referring to FIG. 2E, it shows different processed images generated by thresholding the entire fluorescence image, called the thresholded (fluorescence) image 225Fw. In particular, the thresholded image 225Fw is obtained by dividing the fluorescence image into a (foreground) target segment and a (background) non - target segment, and its fluorescence values are above and below a threshold calculated according to the fluorescence values (e.g., minimizing the inter - class variance), and the target segment representing the tumor is emphasized (e.g., colored) with respect to the non - target segment (e.g., white and black) representing the remaining part of the surgical cavity different from the tumor.

[0046] The figure also shows a processed image similarly generated by only threshold processing the information region 210Fi, which is referred to as the threshold processed (fluorescence) image 225Fi. In the threshold processed image 230Fw, the wide statistical distribution of its fluorescence values raises the threshold (due to its higher fluorescence values). This classifies the fluorescence values within the information region 210Fi into non-target segments, thereby eliminating the tumor. In the threshold processed image 225Fi, instead, the fluorescence values within the information region 210Fi are more accurately classified due to the lower threshold (due to their narrower statistical distribution). This enables the identification of the tumor 112 within the information region 210Fi.

[0047] The above approach facilitates the identification of tumors (or any other target body). For example, the risk of over-detection (false positives) of tumors, especially the risk of under-detection (false negatives) of tumors, is significantly reduced. This avoids (or at least significantly reduces) the over-removal of healthy tissue, especially the incomplete resection of tumors, and furthermore, this significantly improves the detection of tumor lesions. All of the above have beneficial effects on the health of the patient.

[0048] Referring now to FIG. 3, it shows the main software components that can be used to implement the approach according to an embodiment of the present disclosure.

[0049] All software components (programs and data) are shown generally as reference 300. The software component 300 is typically stored in a mass memory and, when the program is executed, is (at least partially) loaded into the working memory of the central unit of the imaging system, together with the operating system and other application programs not directly related to the approach of the present disclosure (omitted in the figure for simplicity). The program is initially installed in the mass memory, for example, from a removable storage unit or from a communication network. In this regard, each program may be a module, segment, or part of the code, which includes one or more executable instructions for implementing a specific logical function.

[0050] The fluorescence drive unit 305 drives the fluorescence unit (including an excitation light source and a fluorescence camera) of a dedicated imaging system to acquire a fluorescence image of a properly illuminated field of view for this purpose. The fluorescence drive unit 305 accesses (in write mode) a fluorescence image repository 310, which stores (for assisting the corresponding surgical procedure) a series of fluorescence images continuously acquired during the ongoing imaging process.

[0051] Each fluorescence image is defined by a bitmap including a matrix of cells (e.g., 512 rows and 512 columns), each storing the (fluorescence) value of a basic pixel, i.e., the (fluorescence) value corresponding to the (fluorescence) location of the field of view. Each pixel value defines the brightness of the pixel as a function of the intensity of the fluorescence light emitted at that location and further as a function of the amount of fluorescent agent present there (e.g., ranging from black when no fluorescent agent is present to white as the amount of fluorescent agent increases).

[0052] Similarly, the reflection drive unit 315 drives the reflection unit (including a white light source and a reflection camera) of a dedicated imaging system to acquire a reflection image of a properly illuminated field of view for this purpose. The reflection drive unit 315 accesses (in write mode) a reflection image repository 320, which stores a series of reflection images continuously acquired during the same imaging process (synchronized with the fluorescence images in the corresponding repository 310).

[0053] Each reflection image is defined by a bitmap including a matrix of cells (having the same or a different size for the reflection image), each storing the (reflection) value of a pixel corresponding to the (reflection) location of the field of view. Each pixel value defines the visible light reflected at that location (e.g., its RGB components). A preparator 325 preprocesses the reflection images as needed to prepare them for the subsequent identification of information regions therein.

[0054] The preparator 325 accesses (in read mode) the reflected image repository 320 and (optionally) the fluorescent image repository 310, and accesses (in write mode) the prepared reflected image repository 330. The prepared reflected image repository 330 comprises an entry for each reflected image within the corresponding repository 320. The entry stores the corresponding prepared reflected image if the reflected image is suitable for identifying an information region, and stores a null value otherwise.

[0055] The prepared reflected image is formed by a matrix of cells having the same or a different size than the reflected image, each storing a corresponding (prepared) pixel value. A segmenter 335 (semantically) divides the prepared reflected images into their information and non-information regions. Each prepared reflected image is divided according to its content and, if possible, also according to the content of the corresponding fluorescent image (which can provide additional information that may be potentially useful even if semantically poor).

[0056] The segmenter 335 accesses (in read mode) the prepared reflected image repository 330 and (if necessary) the fluorescent image repository 310, and accesses (in write mode) the reflected segmentation mask repository 340. The reflected segmentation mask repository 340 comprises an entry for each one within the prepared reflected image repository 330. The entry stores the corresponding reflectance segmentation mask for the prepared reflected image, and stores a null value otherwise.

[0057] The reflection segmentation mask is formed by a matrix of cells having the same size as the prepared reflection image, each storing a label indicating the classification of the corresponding pixel, and in this case having two classes (information class and non - information class for information regions and non - information regions). The labels are binary values, for example, which are asserted (activated) (e.g., logical value 1) if the pixel belongs to the information region and de - asserted (de - activated) (e.g., logical value 0) if the pixel belongs to the non - information region.

[0058] The equalizer 345 determines the optical properties of the fluorescent light of the materials appearing in the prepared reflection image limited to those information regions. The equalizer 345 accesses (in read mode) the prepared reflection image repository 330 and the reflection segmentation mask repository 340, and accesses (in write mode) the reflection equalization map repository 350. The reflection equalization map repository 350 has an entry for each one in the prepared reflection image repository 330. The entry stores the corresponding reflection equalization map for the prepared reflection image or a null value if not.

[0059] The reflection equalization map is formed by a matrix of cells having the same size as the prepared reflection image, each storing the (optical) value(s) of the optical parameter(s) (e.g., its reflectance, absorption, etc.) of the fluorescent light of the material appearing in the corresponding pixel. The adapter 355 adapts the reflection segmentation mask and the reflection equalization map to the fluorescent image as needed, making their sizes equal and synchronizing them.

[0060] The adapter 355 accesses (in read mode) the reflection segmentation mask repository 340, the reflection equalization map repository 350, and the fluorescent image repository 310, and it accesses (in read / write mode) the fluorescent segmentation mask repository 360 and the fluorescent equalization map repository 365. The fluorescence segmentation mask repository 360 comprises a fluorescence segmentation mask for each fluorescence image within the corresponding repository 310.

[0061] The fluorescence segmentation mask is formed by a matrix of cells having the same size as the fluorescence image, each storing the label of the corresponding pixel as described above (i.e., asserted or de-asserted depending on whether the pixel belongs to the information region or the non-information region). The fluorescence colocalization map repository 365 comprises a fluorescence colocalization map for each fluorescence image within the corresponding repository 310. The fluorescence colocalization map is formed by a matrix of cells having the same size as the fluorescence image, each storing the optical value of the corresponding pixel.

[0062] The processor 370 processes (e.g., by auto-scaling and / or thresholding) the fluorescence images limited to their information regions to facilitate the identification of the manifestation of tumors therein. The processor 370 accesses (in read mode) the fluorescence image repository 310, the fluorescence segmentation mask repository 360, and the fluorescence colocalization map repository 365, and accesses (in write mode) the processed fluorescence image repository 375. The processed fluorescence image repository 375 comprises a processed fluorescence image for each fluorescence image within the corresponding repository 310. For example, the processed fluorescence image is an auto-scaled fluorescence image (in the case of auto-scaling) or a thresholded fluorescence image (in the case of thresholding).

[0063] The processed fluorescence image is formed by a matrix of cells having the same size as the fluorescence image, each storing a corresponding pixel (processed, i.e., automatically scaled / thresholded) value. A visualizer 380 generates an output image based on the processed fluorescence image for visualization purposes. The visualizer 380 accesses (in read mode) the processed fluorescence image repository 375 and (optionally) the fluorescence mask repository 360 and the reflected image repository 320, and accesses (in write mode) the output image repository 385. The output image repository 385 comprises output images for each processed fluorescence image within the corresponding repository 3752. For example, the output image is equal to only the processed fluorescence image or the processed fluorescence image overlaid on the corresponding reflected image. A monitor driver 390 drives the monitor of the imaging system to display the output image (substantially in real time during a surgical procedure). The monitor driver 390 accesses (in read mode) the output image repository 385.

[0064] Referring now to FIGS. 4A - 4B and 5, various operation diagrams are shown depicting the flow of operations associated with the implementation of a technique according to an embodiment of the present disclosure. In this regard, each block may correspond to one or more executable instructions for implementing a particular logical function on a corresponding computer device.

[0065] Starting with FIGS. 4A - 4B, these operation diagrams represent an exemplary process that can be used to image a patient using method 400. This process is executed on the central unit of the imaging system during a surgical procedure on the patient. A technique according to an embodiment of the present disclosure (where the identification of the information region in the reflected image is transferred to the corresponding fluorescence image to facilitate the identification of tumors therein) can be applied either indiscriminately (always) or selectively (e.g., in response to a corresponding command (e.g., input by pressing a dedicated button of the imaging system) by activating / deactivating it).

[0066] Before a surgical procedure (e.g., a few days before), a health management operator (e.g., a nurse) administers a fluorescent agent to the patient. The fluorescent agent (e.g., indocyanine green, methylene blue, etc.) is suitable for reaching a specific (biological) target body, e.g., a tumor to be excised. This can be achieved by using either a non-targeted fluorescent agent (e.g., adapted to accumulate in the target body without specific interactions such as passive accumulation) or a targeted fluorescent agent (adapted to attach to the target body using specific interactions, e.g., by incorporating a target-specific ligand into the formulation of the fluorescent agent based on chemical binding properties and / or physical structures adapted to interact with various tissue, vascular, metabolic properties, etc.).

[0067] The fluorescent agent is administered intravenously to the patient as a bolus (using a syringe), as a result of which the fluorescent agent circulates within the patient's vascular system until it reaches the tumor and binds to it. Alternatively, the remaining (unbound) fluorescent agent is cleared from the blood pool (according to the corresponding half-life). After a waiting time (e.g., from a few minutes to 24 - 72 hours) during which the fluorescent agent accumulates in the tumor and is washed away from other parts of the patient's body, the surgical procedure can be started. At this point, the operator brings an imaging probe close to the area of the patient where the surgical cavity is opened by the surgeon, and the operator enters a start command into the imaging system (e.g., its keyboard).

[0068] In response, the imaging process starts by proceeding from the black starting circle 402 to block 404. At this point, the fluorescence drive unit and the reflection drive unit turn on the excitation light source and the white light source respectively to illuminate the field of view. Then, the flow of operations branches into two operations that are carried out simultaneously. In particular, in block 406, the fluorescence drive unit acquires a (new) fluorescence image and adds it to the corresponding repository. At the same time, in block 408, the reflection drive unit acquires a (new) reflection image and adds it to the corresponding repository. In this way, the fluorescence image and the reflection image are acquired almost simultaneously, and they provide different representations (in the sense of fluorescence and visible light respectively) of the same field of view that are spatially coherent. (That is, a predictable correlation exists between these pixels and reaches perfect identity).

[0069] From blocks 406 and 408 to block 410, the flow of operations converges again, and the preparator searches the reflection image that was just added to the corresponding repository and preprocesses it for the next identification of the information area in it if necessary. For example, the preparator can verify whether the reflection image is suitable for identifying the information area. For this purpose, the average and / or variance of its pixel values can be calculated.

[0070] If the average is lower (possibly strictly) than a certain (dark) threshold (meaning that the reflection image is too dark), and / or if the variance is higher (possibly strictly) than a certain (blurry) threshold (meaning that the reflection image is too blurry), the quality of the reflection image is not considered acceptable to provide meaningful identification information for the information area. In this case, the preparator ignores the reflection image by adding a null value to the prepared reflection image repository.

[0071] Conversely (meaning that the quality of the reflected image is acceptable and the identification of the information area there is possible), the preparer can apply one or more filters to further improve the quality of the reflected image (e.g., color normalization, noise reduction, illumination correction, distortion reduction, reflection removal, etc.). In particular, when the average is higher (preferably exactly) than the dark threshold and lower than the higher (luminance) threshold (meaning that the reflected image is not very bright), for example, equal to 1.2 to 1.5, the preparer can apply histogram equalization to the reflected image (by spreading the most frequent pixel values, a nearly flat histogram can be obtained). In fact, the experimental results show that histogram equalization improves performance in this case, while it may degrade performance otherwise.

[0072] Additionally or alternatively, the preparer may downscale the reflected image to reduce the computational complexity (e.g., perform subsampling following low-pass filtering). Additionally or alternatively, the preparer may group the pixels of the reflected image into substantially homogeneous groups, each of which is represented by a group value based on the corresponding pixel values, simplifying the identification of the information area (e.g., by applying clustering, graph-based, random walk, watershed edge detection, and similar algorithms).

[0073] Additionally or alternatively, the preparer may apply a motion compensation algorithm (to align the reflected image with the fluorescence image) and / or apply a warping algorithm (to correct the distortion of the reflected image with respect to the fluorescence image). In either case, the preparer adds the thus obtained prepared reflected image (equal to the corresponding reflected image if possible) to the corresponding repository.

[0074] The flow of operations branches in block 412 according to the content of the entry immediately added to the prepared reflection image repository. If the entry contains a (prepared) reflection image, the segmenter extracts this reflection image from the corresponding repository for its semantic segmentation in block 414. In computer vision, semantic segmentation is a specific type of segmentation (generally aimed at dividing an image into separate parts or segments with more or less homogeneous characteristics), and the segments represent entities belonging to various classes with corresponding meanings (i.e., concepts that abstract the common characteristics of their multiple instances). In this particular case, semantic segmentation aims to divide the reflection image into an information area representing the surgical cavity without foreign objects and a non-information area representing foreign objects (i.e., surgical instruments, hands, surgical tools, surrounding body parts, and / or background materials). And the flow of operations branches in block 416 according to the implementation of the segmenter. In particular, if the segmenter is based on a classification algorithm, blocks 418 - 422 are executed, while if the segmenter is based on a deep learning approach, block 424 is executed.

[0075] Referring now to block 418 (classification algorithm), the segmenter performs a feature extraction step to extract one or more features from the reflection image and, if possible, also from the corresponding fluorescence image (which has been pre-judged to be most suitable for this purpose, as will be described later). Each feature is a (measurable) characteristic representing a unique feature of the reflection / fluorescence image. Examples of these features are saturation, hue, luminance, Histogram of Oriented Gradients (HOG), variance, Bag Of Visterms (BOV), Scale-Invariant Feature Transform (SIFT), etc.

[0076] More specifically, the segmenter calculates one or more feature maps, each of which is formed by a matrix of cells having the same size as the reflection / fluorescence image, and each stores the (feature) value of the corresponding feature. For this purpose, the segmenter applies the corresponding filters to the reflection / fluorescence image (e.g., smoothing (e.g., Gaussian blur, Kuwahara, anisotropic diffusion, etc.), statistics (e.g., mean, median, entropy, etc.), edge detectors (e.g., Sobel, Prewitt, Canny, etc.), derivatives, Hessian matrix, Laplacian, etc.). Each filter calculates the feature value at each location according to the corresponding pixel value and, if possible, takes into account the pixel values of its neighboring pixels.

[0077] In block 420, the segmenter calculates a reflection segmentation mask corresponding to the reflection image by applying a specific classification algorithm to the feature map and adds it to the corresponding repository. For example, the classification algorithm is a conditional random field (CRF) algorithm. Basically, the CRF algorithm calculates the label of each pixel by an inference step that determines the value of the label that maximizes the posterior probability that the pixel belongs to the corresponding class. The posterior probability is based on a node (or unary) potential that depends only on the feature value of the pixel and an edge (or pairwise) potential that takes into account its neighboring pixels (either these labels that smooth the transition between segments or these feature values that model similarities).

[0078] The segmenter enhances, if necessary, the thus obtained reflection segmentation mask in block 422. For example, the segmenter may perform a fill hole (hole filling) step, where any severed parts of the non-information area, i.e., parts completely surrounded by the information area, are assigned to the information area (assuming that the foreign object is not completely surrounded by the surgical cavity). Additionally or alternatively, the segmenter can perform one or more smoothing steps to remove isolated misclassified pixels (e.g., by applying erosion, dilation, box filter convolution, and similar algorithms).

[0079] Referring instead to block 424 (deep learning), the segmenter is a (artificial) neural network, e.g., a U-Net (appropriately trained for this purpose, as will be described later). Basically, deep learning is a specific type of machine learning (used to perform a specific task, which in this case is to semantically segment a reflection image without using explicit instructions, but inferring from the examples how to do it automatically), which is based on a neural network.

[0080] A neural network is a data processing system that approximates the operation of the human brain. A neural network includes basic processing elements (neurons), which perform operations based on corresponding weights. The nodes are connected via one-way channels (synapses), which transfer data between them. Neurons are organized into layers that perform various operations and always include an input layer and an output layer for receiving input data and providing output data, respectively (in this case, the reflection image preferably comprises a corresponding fluorescence image and a corresponding reflection segmentation mask).

[0081] A deep neural network (DNN) is a specific type of neural network with one or more (hidden) layers between an input layer and an output layer. A convolutional neural network (CNN) is a specific type of deep neural network in which one or more of those layers perform (mutual) convolution operations. In particular, a CNN includes one or more convolutional layers that calculate corresponding feature maps and one or more pooling layers that reduce their resolution, and one or more fully connected layers segment the fluorescence image according to these (reduced) feature maps.

[0082] U-net is a specific convolutional neural network, where an expansion path follows the convergence path (formed by convolutional and pooling layers), and conversely, the expansion path includes one or more upsampling layers that increase the resolution of the feature maps and subsequently one or more convolutional layers that assemble them, and there are no fully connected layers (providing a U-shaped architecture with an expansion path that is approximately symmetric to the convergence path). In this case, the segmenter (which receives the reflection / fluorescence image) directly generates a reflection segmentation mask and adds it to the corresponding repository.

[0083] In both cases, the flow of operation reunites at block 426 from block 422 or block 424. At this point, the equalizer determines the optical properties of the reflection segmentation mask that was just added to its repository and the corresponding (prepared) reflection image from its repository to limit the information area of the reflection image.

[0084] Therefore, the equalizer considers each pixel of the reflected image. If the corresponding label in the reflection segmentation mask is asserted (which means the pixel belongs to the information region), the equalizer determines the type of the biological material represented by the corresponding pixel value (e.g., blood, muscle, fat, etc. according to the color of the pixel value), and adds its optical value (e.g., in the range from 0 to 1 according to the type of the biological material and the luminance of the pixel value) to the corresponding cell of the reflection equalization map. On the other hand, if the corresponding label in the reflection segmentation mask is de-asserted (which means the pixel belongs to the non-information region), the equalizer adds a null value to the corresponding cell of the reflection equalization map.

[0085] In block 428, the adapter searches for the reflection segmentation mask and the reflection equalization map that have been immediately added to the corresponding repository, and adapts them to the corresponding fluorescence image (from the corresponding repository) if necessary. For example, the adapter can resize / enlarge the reflection segmentation mask and the reflection equalization map to adapt them to the fluorescence image when they have various sizes (e.g., using low-pass filtering followed by subsampling, or interpolation followed by low-pass filtering). In either case, the adapter adds the fluorescence segmentation mask and the fluorescence equalization map (equal to the reflection segmentation mask and the reflection equalization map respectively and adapted to the fluorescence image if possible) to the corresponding repository.

[0086] Referring back to block 412, if the entry just added to the prepared reflection image repository contains a null value, the process instead descends to block 430. In this case, the adapter estimates a fluorescence segmentation mask and a fluorescence icolization map corresponding to the missing (prepared) reflection image according to one or more preceding fluorescence segmentation masks and fluorescence icolization maps (extracted from the corresponding repository). For example, each of the fluorescence segmentation mask and the fluorescence icolization map is simply set equal to the preceding one or calculated by interpolating two or more preceding ones. As described above, the adapter adds such obtained fluorescence segmentation mask and fluorescence icolization map to the corresponding repository.

[0087] In either case, the flow of operation rejoins at block 432 from block 428 or block 430. At this point, the processor searches the fluorescence segmentation mask and fluorescence icolization map just added to those repositories and the corresponding fluorescence image from that repository to (post) process the fluorescence image restricted to its information region (and possibly also the one restricted to its non-information region). For example, the first of all the processors equalizes the fluorescence image (restricted to its information region) according to the corresponding optical characteristics of the reflection image if necessary.

[0088] For this purpose, the icolizer considers each pixel of the fluorescence image and, if the corresponding label in the fluorescence segmentation mask is asserted (meaning that the pixel belongs to the information region), the icolizer updates the pixel value according to the corresponding optical value in the fluorescence icolization map (e.g., by increasing it if the optical value indicates that the corresponding biological material such as blood significantly shields fluorescence).

[0089] In this way, it is possible to compensate for the influence of various biological materials on the acquired fluorescence light. In particular, the limitation of the operation to the information area only avoids the foreign matter from having an adverse effect on the result. The flow of the operation branches in block 434 according to the type of processing applied to the (possibly equalized) fluorescence image. In particular, in the case of auto-scaling, blocks 436 to 446 are executed, while in the case of threshold processing, blocks 448 to 464 are executed. In both cases, the flow of the operation reunites at block 466.

[0090] Referring now to block 436 (auto-scaling), the processor determines the (information) fluorescence range of the information area as the difference between its highest pixel value and its lowest pixel value, and the (non-information) fluorescence range of the non-information area as the difference between its highest pixel value and its lowest pixel value.

[0091] The information fluorescence range and the predefined display range of the monitor (searched from the corresponding configuration variable) are used as parameters of a predefined parametric function (e.g., logarithmic) to obtain the corresponding information mapping function. Similarly, the non-information fluorescence range and the display range are used as parameters of a predefined parametric function to obtain the corresponding non-information mapping function (which may be the same as or different from the above, e.g., by adding a scaling factor to mask the content of the non-information area).

[0092] Then, to auto-scale the information area and the non-information area separately, a loop is entered. The loop starts at block 438, and the processor considers the (current) pixel of the fluorescence image (starting from the first one in any order). The process branches in block 440 according to the corresponding label in the fluorescence segmentation mask. If the label is asserted (meaning the pixel belongs to the information area), the processor converts the corresponding pixel value by applying the information mapping function in block 442 and adds it to the same cell of the (new) auto-scaled fluorescence image (to the corresponding temporary variable).

[0093] Conversely, when the label is de-asserted (meaning that the pixel belongs to the non-information area), in block 444, the processor applies a non-information mapping function to convert the corresponding pixel value and add it to the same cell of the auto-scaled fluorescence image. In both cases, in block 446, the processor verifies whether the last pixel of the fluorescence image has been processed. If not, the process returns to block 438 and repeats the same operation for the next pixel. Conversely (once all pixels have been processed), the loop ends by descending into block 466.

[0094] Referring instead to block 448 (thresholding), the processor determines the threshold using only the pixel values of the information area (e.g., by applying Otsu's algorithm to this). Then it enters a loop to threshold the information area. The loop starts at block 450 and the processor considers the (current) pixel of the fluorescence image (starting from the first one in any order).

[0095] The process branches in block 452 according to the corresponding label in the fluorescence segmentation mask. When the label is asserted (meaning that the pixel belongs to the information area), in block 454, the processor compares the corresponding pixel value with the threshold. If the pixel value is higher than (preferably strictly) the threshold (i.e., it belongs to the target segment), in block 456, the processor copies it (to the corresponding temporary variable) to the same cell of the (new) thresholded fluorescence image.

[0096] Conversely, if the pixel value is lower than (or, if possible, exactly) the threshold value (i.e., it belongs to a non-target segment), the processor resets (masks) the pixel value in the same cell of the thresholded fluorescence image to zero in block 458. If the label is deasserted (which means the pixel belongs to a non-informative area), the same point is reached from block 452. In either case, the process continues from either block 456 or block 458 to block 460. At this point, the processor verifies whether the last pixel of the fluorescence image has been processed. If not, the process returns to block 450 and repeats the same operation for the next pixel.

[0097] Conversely (once all pixels have been processed), the loop ends by descending to block 462. Alternatively, the processor generates a thresholding mask (formed by a matrix of cells of the same size as the fluorescence image, each storing a flag). For each pixel value of the fluorescence image, if both corresponding labels in the fluorescence segmentation mask are asserted and the pixel value is higher than (or, if possible, exactly) the threshold value, the processor asserts the corresponding flag (e.g., to value 1), otherwise it deasserts the corresponding flag (e.g., to value 0).

[0098] Referring now to block 462, the processor can further process the thresholded fluorescence image thus obtained. For example, the processor calculates one or more (target) statistical parameters of the target segment and one or more (non-target) statistical parameters of the non-target segments of the thresholded fluorescence image (e.g., their mean and standard deviation).

[0099] For this purpose, considering the case of using a threshold mask (along with similar considerations applied otherwise), the processor, considering each pixel of the fluorescence image, if the corresponding flag within the threshold mask is asserted (which means the pixel belongs to the target segment), the processor uses the corresponding pixel value to increase the calculation of the target statistical parameter, while if the corresponding flag within the threshold mask is de-asserted and the label within the fluorescence segmentation mask is asserted (which means the pixel belongs to the non-target segment), the processor uses the corresponding pixel value to increase the calculation of the non-target statistical parameter.

[0100] The processor updates the pixel values of the target segment in block 464 according to the target statistical parameter, the non-target statistical parameter, or both. For this purpose, the processor, considering each pixel of the thresholded fluorescence image, for example, if the corresponding flag within the thresholding mask is asserted (which means the pixel belongs to the target segment), the processor subtracts the average of the non-target segment from the corresponding pixel value and divides the resulting value or (the original) pixel value by the standard deviation of the target segment, by the standard deviation of the non-target segment, or by a combination thereof (e.g., their sum, difference, average, etc.). Then the process descends into block 466.

[0101] Referring now to block 466, the processor adds the thus-obtained processed fluorescence image to the corresponding repository (and, possibly, adds the corresponding threshold processing mask, if any, to another repository). The visualization device, in block 468, retrieves from the corresponding repositories the processed fluorescence image added to the corresponding repository and, optionally, the corresponding fluorescence segmentation mask and reflection image (and, possibly, the corresponding threshold processing mask). The visualization device generates a corresponding output image (based on the processed fluorescence image) and adds it to the corresponding repository. For example, the visualization device may simply set an output image that is simply equal to the processed fluorescence image only (in the case of the threshold processing mask, the same result is achieved by masking the pixel values of the fluorescence image where the corresponding flag in the threshold processing mask is deasserted).

[0102] Additionally or alternatively, the visualization device may generate an integrated image given by the pixel values of the processed fluorescence image and the reflection image in the information region and the non-information region (after rescaling the processed fluorescence image or the reflection image and equalizing their sizes, if necessary). Alternatively, an overlay image given by the pixel values of the processed fluorescence image greater than zero (or where the flag in the threshold processing mask is asserted) and the pixel values of the reflection image may be generated. In either case, the monitor driver 390, in block 470, displays the output image that has been immediately added to the corresponding repository. Thus, the output image is displayed substantially in real time with respect to the acquisition of the corresponding fluorescence / reflection image (apart from a short delay due to their generation).

[0103] Referring now to block 472, if the imaging process is still in progress, the flow of operations returns to before blocks 406 - 408 and continuously repeats the same operations. Conversely, if the imaging process has ended, the process ends with a concentric white / black stop circle 474, as indicated by an end command entered into the imaging system by the operator (e.g., using its keyboard), after turning off the excitation light source and the white light source via the corresponding driver.

[0104] Moving on to FIG. 5, the operation diagram represents an exemplary process that can be used to configure a segmenter using Method 500 (during the development of the imaging system and, if possible, during its subsequent verification). This process is executed on a configuration (computing) system, for example, a personal computer (including one or more microprocessors, non-volatile memory, volatile memory, mass memory, and a controller for its peripherals). For this purpose, the configuration system includes the following software components. The configurator is used to configure the segmenter. The configurator accesses (in read / write mode) a reflection image repository that stores a plurality of (image) sequences of (reference) reflection images, and it accesses (in read / write mode) a reflection segmentation mask repository that stores the corresponding (sample) reflection segmentation masks.

[0105] The process starts at the black start circle 503 and then proceeds to block 506, where the configurator uploads (e.g., via a removable storage unit or a communication network) a plurality of image sequences of reflection images obtained as described above during different (sample) surgical procedures via the corresponding imaging system (e.g., hundreds of image sequences, each having dozens of reflection images) and adds them to the corresponding repository. Following block 509, each reflection image is manually divided into its (reflection) information region and (reflection) non-information region. For example, this result is achieved in a semi-automatic approach, where pre-segmentation is automatically performed (e.g., by applying the SIOX (Simple Interactive Object eXtraction) algorithm), and the resulting result is manually refined to correct possible errors. The resulting reflection segmentation mask is added to the corresponding repository.

[0106] The configurator selects a training set in block 512, which is formed by a part of the (image / mask) pairs of available reflection images and corresponding reflection segmentation masks. For example, the training set is defined by randomly and uniformly sampling the available image / mask pairs among all of them (so that the longer the image sequence, the higher the sampling frequency), or by sampling uniformly within the image sequence (so that the same number of image / mask pairs are provided for each image sequence). The flow of operations branches in block 515 according to the implementation of the segmenter. In particular, when the segmenter is based on a classification algorithm, blocks 518 - 521 are executed, while when the segmenter is based on a deep learning method, block 524 is executed.

[0107] Referring now to block 518 (classification algorithm), the configurator executes a feature selection step that determines an (optimization) set of features that optimize the performance of the segmenter among a number of possible (candidate) features. For example, the feature selection step is based on a wrapper method where the optimization set is determined using iterative optimization of the classification algorithm. For this purpose, a brute force approach is first applied to initialize the optimization set.

[0108] To make the computational complexity of the operations actually executable, the initialization of the optimization set is limited to a maximum size of several units (e.g., ≤ 3), and a simplified version of the classification algorithm is used (e.g., in the case of the CRF algorithm, it is limited to node potentials with a default model based on, for example, the Naive Bayes algorithm). For this purpose, the configurator considers all (feature) combinations of candidate features formed by a number equal to at most the maximum size.

[0109] For each feature combination, the configurator causes the segmenter to calculate a reflection segmentation mask corresponding to the reflection images of the training set by applying a classification algorithm using this feature combination. The configurator calculates a quality metric indicating the quality of the segmentation provided by the feature combination.

[0110] For this purpose, the configurator calculates a similarity metric that measures the similarity between each (calculated) reflection segmentation mask thus obtained and the corresponding (reference) reflection segmentation mask in the training set. For example, the similarity index is defined as twice the number of pixels having the same label in the calculated / reference reflection segmentation mask divided by the total number of pixels (in the range from 0 to 1 in increasing order of similarity), by the Sørensen-Dice coefficient, or is defined by metrics such as Jaccard, Bray-Curtis, Czekanowski, Steinhaus, Pielou, Hellinger, etc.

[0111] Then, the segmenter calculates the quality metric as the average of all similarity metrics of the calculated reflection segmentation mask with respect to the corresponding reference reflectance segmentation mask. The optimized set is initialized with the feature combination that provides the highest quality metric. Then, the configurator applies a step-by-step approach to expand the optimized set. For this purpose, the configurator takes into account all additional candidate features that are not already included in the optimized set.

[0112] For each additional (candidate) feature, the configurator causes the segmenter to calculate a reflection segmentation mask corresponding to the reflection images of the training set by applying a classification algorithm using the features of the optimized set and this additional feature, and then it calculates the corresponding quality metric as described above.

[0113] If the (best) additional feature (which provides the highest quality metric when added to the optimized set) includes a significant improvement (e.g., the difference between the quality metric of the optimized set with the best additional feature and the quality metric of the (original) optimized set is higher than a minimum value, e.g., higher than 5 - 10%, and as exact as possible if possible), the configurator adds the best additional feature to the optimized set. The same operation is repeated until an acceptable quality metric is obtained, the best additional feature no longer provides a significant improvement, or the optimized set reaches the maximum allowable size (e.g., 10 - 15).

[0114] In block 521, the configurator optionally selects one or more operation parameters of the classification algorithm that optimize the performance of the segmenter. For example, in the case of the CRF algorithm, this includes the selection of an optimized node model and an optimized edge model for calculating the node potential and the edge potential respectively, and the selection of optimized values for the (node) model parameters and the (edge) model parameters.

[0115] To make the computational complexity of the operation actually executable, the selection of the optimized node / edge model is performed by an empirical approach independently of the feature selection step (i.e., by using the optimized set determined as described above). For example, first the configurator maintains an edge model fixed to a default value (e.g., the Potts model) and selects the optimized node model from among a plurality of possible (candidate) node models (e.g., based on Naive Bayes, Gaussian mixture model, k - nearest neighbor method, artificial neural network, support vector machine, and similar algorithms) by using the default values of the node / edge model parameters.

[0116] For this purpose, for each candidate node model, the configurator causes the segmenter to calculate a reflection segmentation mask corresponding to the reflection images of the training set by applying a classification algorithm using this candidate node model, which calculates the corresponding quality metric as described above. The configurator sets the optimized node model to the candidate node model that provides the highest quality metric.

[0117] Thereafter, the configurator selects an optimized edge model from among a plurality of possible (candidate) edge models (e.g., based on Potts, contrast-sensitive Potts, prior probability, and contrast-sensitive Potts models with similar algorithms such as Potts) using the default values of the node / edge model parameters (e.g., for the classification algorithm using the optimized node model determined above). For this purpose, for each candidate edge model, the configurator causes the segmenter to calculate a reflection segmentation mask corresponding to the reflection images of the training set by applying a classification algorithm using this candidate edge model, which calculates the corresponding quality metric as described above.

[0118] The configurator sets the optimized edge model to the candidate edge model that provides the highest quality metric. Finally, the configurator searches for optimized values of the node / edge model parameters for the optimized node model and optimized edge model determined above. For this purpose, the configurator causes the segmenter to apply a classification algorithm using the optimized node / edge model and vary its model parameters to calculate a reflection segmentation mask corresponding to the reflection images of the training set, which calculates the corresponding quality metric as described above. The operation is driven by an optimization method (e.g., using the Powell search algorithm) until an acceptable quality metric is obtained.

[0119] Instead, referring to block 524 (deep learning), the configurator uses the training set to perform training steps of the neural network of the segmenter in order to find the (optimized) values of its weights that optimize the performance of the segmenter. To make the computational complexity of the operations actually feasible, the training steps are based on an iterative process, for example, based on the Stochastic Gradient Descent (SGD) algorithm. For this purpose, first, the configurator initializes the weights of the neural network (e.g., randomly).

[0120] The configurator inputs the reflected images of the training set into the neural network, obtains the corresponding reflected segmentation masks, and it calculates the corresponding quality metrics as described above. The configurator determines the changes in the weights that should improve the performance of the neural network. In particular, in the SGD algorithm, the direction and amount of the change are given by the gradient of the error function with respect to the weights, which is approximated by the backpropagation algorithm. The same operations are repeated until an acceptable quality metric is obtained or the change in the weights no longer provides a significant improvement (i.e., it means that at least a local or flat region of the minimum of the error function has been found).

[0121] The weights may be changed either in iterative mode (after obtaining all the reflected segmentation masks) or in batch mode (after obtaining all the reflected segmentation masks). In either case, the weights are changed using the addition of random noise and / or the training steps are repeated, starting with one or more different initializations of the neural network to find different (and if possible better) minima and identify flat regions of the error function. Thus, the features (implicitly defined by the weights) used to segment the reflected images are automatically determined during the training steps without the need for their explicit selection.

[0122] In both cases, the flow of operations converges at block 527 from either block 521 or block 524. At this point, the configurator selects a test set, which is formed by a part of the image / mask pairs defined by sampling the available image / mask pairs randomly and uniformly among all of them, or homogeneously within the image sequence. The configurator causes the segmenter thus obtained to calculate, at block 530, a reflection segmentation mask corresponding to the reflection image of the test set, which calculates the corresponding quality metric as described above. The flow of operations branches at block 533 according to the quality metric.

[0123] If the quality metric is lower than (strictly, if possible) the tolerance value, this means that the generalization ability of the segmenter (from the configuration based on the training set to the test set) is too poor. In this case, the process returns to block 512 and repeats the same operation using a different training set (or returns to block 506 (not shown) to enhance the image sequence of the reflection images). Conversely, if the quality metric is higher than (strictly, if possible) the tolerance value, this means that the generalization ability of the segmenter is satisfactory. In this case, the configurator accepts the configuration of the segmenter at block 536 and deploys it to the batch of the imaging system. Then, the process ends with the concentric white / black stop circle 539.

[0124] (Variant) Of course, in order to meet local and specific requirements, those skilled in the art can apply many logical and / or physical modifications and changes to this disclosure. More specifically, although this disclosure has been described in some detail with reference to one or more of its embodiments, it should be understood that forms, details, and various omissions, substitutions, and changes in other embodiments are possible. In particular, various embodiments of this disclosure can be implemented without the specific details (e.g., numerical values) described in the foregoing description, which can provide a more complete understanding. Conversely, well-known features may be omitted or simplified in order not to obscure the description of unnecessary details. Furthermore, it is explicitly intended that the specific elements and / or method steps described in connection with any embodiment of this disclosure can be incorporated as a matter of general design choice in any other embodiment. Moreover, items presented in the same group and various embodiments, examples, or alternatives should not be construed as being in fact equivalent to each other (however, they are separate and autonomous entities). In any case, each numerical value should be read as being subject to correction according to the applicable tolerance, and in particular, unless otherwise indicated, the terms "substantially", "about", "approximately" should be understood to be within a range of 10%, preferably within a range of 5%, and even more preferably within a range of 1%. Furthermore, each range of numerical values should be intended to explicitly specify any possible number along the continuum within the range (including its endpoints). Ordinal numbers or other modifiers are simply used as labels to distinguish elements with the same name and do not, in themselves, imply precedence, priority, or order.The terms "include", "comprise", "have", "contain", "involve", etc. are intended to have an open and non-exhaustive meaning (i.e., not limited to the items described), the terms "based on", "dependent on", "according to", "function of", etc. are intended to be non-exhaustive relationships (i.e., additional possible variables are included), the term "a / an" is intended to mean one or more items (unless otherwise explicitly stated), and the term "means for" (or any means-plus-function form) is intended to mean any structure adapted or configured to perform the associated function.

[0125] For example, one embodiment provides a method for assisting in a patient's medical treatment. However, this method can be used to assist in any medical treatment of any patient (e.g., human, animal, etc.) (e.g., surgical treatment, diagnostic treatment, therapeutic treatment, etc.). Also, the corresponding steps can be performed in any manner (e.g., continuously during a medical treatment, upon request, etc.). In any case, this method can facilitate a physician's task, but it remains a data processing method that only provides information that can assist the physician and is not a medical act in the strict sense that is always performed by the physician himself / herself.

[0126] In one embodiment, this method includes the following steps under the control of a computer device. However, the computer device can be of any type (see below).

[0127] In one embodiment, this method includes (by a computer device) obtaining a luminescent image of the field of view. However, the luminescent image can be obtained in any manner (e.g., directly obtained by controlling any acquisition unit at any frequency, with any excitation light, transferred by a removable storage unit, uploaded via a network, etc.).

[0128] In one embodiment, the field of view includes the patient's region of interest (for medical treatment). However, the region of interest may be of any type (e.g., a surgical cavity for a surgical procedure, an internal cavity for an endoscopic procedure, an open type accessed through a lumen, or a closed type accessed through an incision, etc.).

[0129] In one embodiment, the region of interest includes at least one target body for a medical treatment. However, the number of target bodies may be any number and may be of any type (e.g., a lesion to be resected, identified, monitored or treated, such as a tumor, polyp, inflammation, thrombus, etc., a body part to be treated, such as a bleeding blood vessel to be cauterized, a stenotic esophagus to be dilated, etc., a structure surrounding any item to be operated on by a physician).

[0130] In one embodiment, the target body includes a luminescent substance. However, the luminescent substance may be of any exogenous / endogenous type (e.g., any luminescence phenomenon, such as any luminescent agent based on fluorescence, phosphorescence, chemiluminescence, bioluminescence, stimulated Raman emission, etc., any natural luminescent component).

[0131] In one embodiment, the field of view includes one or more foreign objects that are different from the region of interest. However, the number of foreign objects may be any number and may be of any type (e.g., instruments, hands, tools, body parts, background materials, etc.).

[0132] In one embodiment, the luminescent image includes a plurality of luminescence values that represent luminescence light emitted by a luminescent substance at corresponding luminescent locations in the field of view. However, the luminescent image may have any size and shape (from the entire matrix to one or more of its parts), and may include any type of luminescence value and any luminescent location (e.g., for pixels, voxels, groups thereof, grayscale or color values in RGB, YcBcr, HSL, CIE-L*a*b, Lab color, similar representations). The luminescence light may be of any type (e.g., NIR, infrared (IR), visible light, etc.) and may be emitted in any manner (e.g., in response to corresponding excitation light, or more generally any other excitation different from heating).

[0133] In one embodiment, the method includes acquiring (by a computer device) an auxiliary image of the field of view. However, the auxiliary image may be acquired in any manner (e.g., in the same or different manner as the luminescent image, simultaneously with the luminescent image, or in short succession, etc.).

[0134] In one embodiment, the auxiliary image includes a plurality of auxiliary values that represent auxiliary light (different from the luminescence light) received from corresponding auxiliary locations in the field of view. However, the auxiliary image may have any size and shape, and may include any type of auxiliary value and any auxiliary location (the same or different location as the luminescent image). The auxiliary light may be of any type different from the luminescence light of the luminescent image (e.g., visible light, IR light, ultraviolet (UV) light, other luminescence light of different wavelengths, etc.).

[0135] In one embodiment, the method includes (by a computer device) identifying an auxiliary information region of an auxiliary image that represents a region of interest without foreign objects according to the content of the auxiliary image. However, the auxiliary information region may be of any type (e.g., a single region, one or more separate regions, a corresponding mask, or something directly defined within the auxiliary image, etc.), and it may be identified in any way (e.g., by semantically / non-semantically splitting the auxiliary image into an information region and a non-information region, by searching for the information region within the auxiliary image, etc.).

[0136] In one embodiment, the method includes (by a computer device) identifying a light emission information region of a light emission image corresponding to the auxiliary information region. However, the light emission information region may be identified in any way (e.g., by transferring the identification of the auxiliary information region directly or with any adaptation, etc.). Further, this operation may be performed indiscriminately or may be conditional on the quality of the identification of the auxiliary information region. For example, it is possible to calculate a quality metric, and if the quality metric has not reached the corresponding threshold, it is possible to assign all light emission locations to the light emission information region.

[0137] In one embodiment, the method includes (by a computer device) generating a processed light emission image. However, the processed light emission image may be of any type (e.g., an auto-scaled fluorescence image, a threshold-processed fluorescence image, a segmented fluorescence image, etc.).

[0138] In one embodiment, the processed light emission image is generated by processing a light emission image restricted to the light emission information region. However, this result may be achieved in any way (e.g., by separately processing the information light emission region and the non-information light emission region with the same or different operations, by processing only the information light emission region while leaving the non-information light emission region unchanged, darkening it, canceling it, etc.).

[0139] In one embodiment, the processing of the luminescent image is based on the luminescence values of the luminescent information regions in order to facilitate the identification of the representation of the target body therein. However, this processing may be based on these luminescence values in any way (e.g., their distribution, range, probability, etc.), and this result can be achieved in any way (e.g., by making the representation of the target body more prominent for manual identification, by automatically identifying the representation of the target body, etc.).

[0140] In one embodiment, this method includes outputting (by a computer device) an output image based on the processed luminescent image. However, the output image may be of any type (e.g., the same processed luminescent image, a processed luminescent image integrated / overlaid with a reflection image, etc.), and may be output in any way (e.g., displayed, printed, transmitted remotely, in real time, or offline).

[0141] Further embodiments provide additional advantageous features and may be entirely omitted in a basic implementation.

[0142] In particular, in one embodiment, this method is for assisting in a surgical procedure on a patient. However, the surgical procedure may be of any type (e.g., for therapeutic purposes, for prophylactic purposes, for aesthetic purposes in a standard surgery, for minimally invasive surgeries such as laparoscopy, arthroscopy, angioplasty, etc.).

[0143] In one embodiment, the step of outputting the output image includes displaying (by a computer device) the output image in substantially real time with respect to the step of acquiring the luminescent image. However, the output image may be displayed in any way (e.g., on any display unit such as a monitor, virtual reality glasses, etc., with any delay resulting from its generation).

[0144] In one embodiment, the region of interest is a surgical cavity of the patient. However, the surgical cavity may be of any type (e.g., a wound, an open body part, etc.).

[0145] In one embodiment, at least a portion of the foreign object overlaps with the region of interest. However, the foreign object may be disposed at any position (e.g., overlapping the region of interest to any extent, surrounding it, spaced apart therefrom, any combination thereof, etc.).

[0146] In one embodiment, the foreign object includes one or more medical devices, one or more hands, one or more medical tools, one or more body parts of a patient that are not the subject of a medical treatment, and / or one or more background materials. However, the foreign object may include any number and type of medical devices (e.g., surgical instruments such as surgical knives, forceps, surgical instruments, manipulators, sampling devices, endoscopic instruments such as polyp resection snares), hands (e.g., the hands of a surgeon, assistant, nurse, etc.), medical tools (e.g., surgical tools such as gauze, retractors, drapes, covers, endoscopic tools such as hemostatic clips, perfusion devices, etc.), body parts that are not the subject (e.g., body parts surrounding the region of interest, muscles, organs irrelevant to the medical treatment, e.g., the liver, etc.), background materials (e.g., operating tables, walls, floors, etc.), or generally partial and different additional things (e.g., biological materials of the patient that interfere with the medical treatment, e.g., fecal residues in colonoscopy, food residues in the stomach in gastroscopy, etc.).

[0147] In one embodiment, the auxiliary image is a reflection image, the auxiliary light is visible light, and the auxiliary value represents visible light reflected at a corresponding auxiliary location in the field of view illuminated by white light. However, the white light (and the corresponding visible light) may be of any type (e.g., non-luminescence light that does not cause a significant luminescence phenomenon with respect to the luminescent substance).

[0148] In one embodiment, the luminescent substance is a luminescent agent that has been pre-administered to the patient before the method is performed. However, the luminescent agent can be of any type (e.g., any target luminescent agent, non-target luminescent agent, etc. based on specific or non-specific interactions), and it can be pre-administered by any method (e.g., using a syringe, infusion pump, etc.) and at any time (e.g., beforehand, immediately before the method is performed, continuously during it, etc.). In any case, this is a data processing method that can be performed independently of the interaction with the patient. Furthermore, the luminescent agent can be administered to the patient in a non-invasive manner (e.g., orally for imaging the gastrointestinal tract, via a nebulizer into the airway, via local spray application or local introduction during a medical procedure), or in any case, without the need for specialized medical expertise or substantial physical intervention (e.g., intramuscular injection) to the patient that involves their health risk.

[0149] In one embodiment, the step of identifying the auxiliary information region includes semantically segmenting the auxiliary image. However, the auxiliary image can be semantically segmented by any method (e.g., using a classification algorithm, neural network, etc.).

[0150] In one embodiment, the auxiliary image is semantically segmented into an auxiliary information region corresponding to at least one region of interest class of the region of interest and an auxiliary non-information region corresponding to one or more foreign object classes of foreign objects. However, the number of region of interest classes and foreign object classes can be any number and of any type (e.g., a single region of interest class for the entire region of interest, a single foreign object class for all foreign objects, a single foreign object class for a corresponding part or group of the region of interest, multiple foreign object classes for corresponding foreign object types or groups thereof, etc.).

[0151] In one embodiment, the step of segmenting the auxiliary image includes (by a computer device) semantically segmenting the auxiliary image using a neural network. However, the neural network may be of any type (e.g., U-Net, convolutional neural network, feed-forward neural network, Radial Basis Function neural network, recurrent neural network, modular neural network, etc.). The neural network may be trained in any manner (e.g., based on stochastic gradient descent, real-time recurrent learning, higher-order gradient descent methods, extended Kalman filtering, and similar algorithms) using any number and type of training pairs (e.g., randomly, homogeneously selected).

[0152] In one embodiment, the step of segmenting the auxiliary image includes (by a computer device) determining one or more feature maps for corresponding features of the auxiliary image. However, the features may be of any number and any type (e.g., selected among partially different additional candidate features by any heuristic (rule of thumb), iterative, filtering, etc. method).

[0153] In one embodiment, each of the feature maps includes corresponding feature values of the auxiliary locations. However, the feature values may be determined in any manner (e.g., using any filter, neural network, encoder, reducer, etc.).

[0154] In one embodiment, the method includes (by a computer device) semantically segmenting the auxiliary image by applying a classification algorithm to the feature values of the feature map. However, the classification algorithm may be of any type (e.g., conditional random field, Markov random field, SIOX, GrabCut, decision tree, k-nearest neighbor method, etc.). The classification algorithm may be configured in any way (e.g., using any method such as any heuristic, iteration, filtering, etc. for any of its parameters).

[0155] In one embodiment, the step of identifying the auxiliary information region includes (by a computer device) assigning the cut-off portions to the auxiliary information region if the auxiliary image includes one or more cut-off portions completely surrounded by the auxiliary information region. However, this fill-hole step may be performed in any way (e.g., indiscriminately, or only for cut-off portions larger than a threshold for either the classification algorithm or a deep learning technique, for the auxiliary image or the emission image).

[0156] In one embodiment, the method includes (by a computer device) preprocessing the auxiliary image before identifying the auxiliary information region. However, the auxiliary image may be subject to any number and type of preprocessing steps (additional ones that are partially different from those described above).

[0157] In one embodiment, the method includes (by a computer device) preprocessing the auxiliary image before its segmentation by applying histogram equalization to the auxiliary image. However, this histogram equalization may be performed in any way (e.g., ordinary histogram equalization, adaptive histogram equalization, contrastive limited adaptive equalization, similar algorithms).

[0158] In one embodiment, histogram equalization is performed in response to the luminance indicators of the auxiliary image included between a dark threshold (indicating the possibility of identifying the auxiliary information area) and a bright threshold (higher than the dark threshold). However, the luminance indicators may be calculated in any way (e.g., average, mode, median, etc.), and the dark / bright thresholds may have any value (e.g., predefined, dynamically determined, etc.). In any case, the possibility of performing histogram equalization indiscriminately (always) is not excluded and is never excluded.

[0159] In one embodiment, the step of identifying the auxiliary information area further includes identifying the auxiliary information area (by a computer device) according to the emission image. However, the emission image may be used in any way to identify the auxiliary information area (e.g., by also inputting the emission image into a neural network, extracting one or more additional feature maps from the emission image for use by a classification algorithm, directly or by weighting to limit the influence of the emission image).

[0160] In one embodiment, the step of generating the processed emission image includes performing auto-scaling of the emission information area according to the emission value of the emission information area (by a computer device). However, the emission information area may be auto-scaled in any way (e.g., mapping to its emission value, logarithmic compression, saturation, etc.).

[0161] In one embodiment, the step of performing auto-scaling of the emission information area includes determining the emission range of the emission value of the emission information area (by a computer device). However, the emission range may be determined in any way (e.g., indiscriminately, by ignoring outliers, etc.).

[0162] In one embodiment, the step of performing auto-scaling on the light emission information region includes (by a computer device) converting the light emission value of the light emission information region according to a mapping function that maps the light emission range to a display range for displaying the light emission image. However, the mapping function may be of any type (e.g., non-linear, such as logarithmic, exponential, linear, etc.).

[0163] In one embodiment, the step of generating the processed light emission image includes (by a computer device) performing threshold processing on the light emission information region according to the light emission value of the light emission information region, thereby dividing the light emission information region into a target segment representing the target body and a non-target segment representing the rest of the region of interest different from the target body. However, the light emission information region may be threshold processed in any way (e.g., based on statistical distribution, entropy, clustering, or object attributes of binary, multi-level, or multi-band types).

[0164] In one embodiment, the step of outputting the output image includes (by a computer device) outputting the output image by emphasizing the target segment with respect to the non-target segment. However, the target segment may be emphasized in any way (e.g., by masking the non-target segment, by representing the target segment in color and the non-target segment in black and white, by increasing and / or decreasing the brightness of the target segment and the non-target segment respectively, etc.).

[0165] In one embodiment, the step of performing threshold processing on the light emission information region includes (by a computer device) determining a threshold according to the statistical distribution of the light emission value of the light emission information region. However, the threshold may be determined in any way (e.g., acting on statistical distributions such as bimodal, unimodal, multimodal, etc.).

[0166] In one embodiment, the step of threshold processing the emission information region includes (by a computer device) assigning each of the emission locations in the emission information region to a target segment or a non-target segment according to a comparison between a corresponding emission value and a threshold. However, the emission locations may be assigned to target / non-target segments according to the threshold in any manner (e.g., by generating a threshold-processed emission image or a threshold-processing mask when the threshold is higher and / or lower).

[0167] In one embodiment, the step of threshold processing the emission information region includes (by a computer device) calculating one or more target statistical parameters of the emission values of the target segments and / or one or more non-target statistical parameters of the non-target segments. However, the target / non-target statistical parameters may be any number, may be none for one of them, and may be of any type (e.g., mean, median, mode, standard deviation, variance, skewness).

[0168] In one embodiment, the step of generating the processed emission image includes (by a computer device) updating the emission values of the target segments according to the target statistical parameters and / or the non-target statistical parameters. However, the target segments may be processed in any manner according to these statistical parameters (e.g., according to only the target statistical parameters, only the non-target statistical parameters, both of these, with additional processing that is partially different from those described above, individually, or in any combination of them).

[0169] In one embodiment, the method includes determining, by a computer device, corresponding optical values of at least one optical parameter related to emission light according to the content for the auxiliary location of the auxiliary image restricted to the auxiliary information area. However, the optical parameters may be of any number and any type (for example, additional optical parameters that are partial and different from those described above), and the corresponding optical values may be determined in any way (for example, individually according to the corresponding auxiliary values by applying any classification algorithm or the like).

[0170] In one embodiment, the step of generating the processed emission image includes equalizing, by a computer device, the emission values of the emission information area according to the optical values. However, the emission information area may be equalized in any way at any time (for example, before and / or after autoscaling / threshold processing it, etc.), for example, by equalizing each emission value only according to the optical value of the corresponding auxiliary location, and further according to the optical value of its adjacent auxiliary location, additional further processing that is partial and different from those described above, individually, or any combination thereof.

[0171] In one embodiment, the luminescent substance is a fluorescent substance (the emission image is a fluorescent image, and the emission value represents the fluorescent light emitted by the fluorescent substance at the corresponding emission location illuminated by the excitation light). However, the fluorescent substance may be of any exogenous / endogenous type (for example, for imaging any pathological tissue, any healthy tissue, etc.).

[0172] Generally, when the same technique is implemented in an equivalent way, similar considerations apply (by using more steps or similar steps with the same functions for some of them, removing some non-essential steps, or adding any further optional steps). Further, the steps may be executed in a different order, simultaneously, or in an interleaved manner (at least partially).

[0173] One embodiment provides a computer program configured to cause a computer device to perform the above method when the computer program is executed by the computer device. One embodiment provides a computer program product, which includes one or more computer-readable storage media collectively storing program instructions, where the program instructions are readable by a computer device and cause the computer device to perform the same method. However, the computer program may be implemented as a stand-alone module, as a plug-in, or directly in an existing software program (such as an imaging system manager). In any case, similar considerations apply when the computer program is configured differently or additional modules or functions are provided. Similarly, the memory structure may be of other types or may be replaced with equivalent entities (physical entities not necessarily composed of physical storage media). The computer program may take any form suitable for use by any computer device (see below), thereby configuring the computer device to perform the desired operations. In particular, the computer program may be in the form of external or resident software, firmware, or microcode (either object code or source code, e.g., compiled or interpreted). Further, it is possible to provide the computer program on any computer-readable storage media. The storage media is any tangible medium (itself different from a transient signal) capable of holding and storing instructions used by a computer device. For example, the storage media may be of electronic, magnetic, optical, electromagnetic, infrared, or semiconductor type. Examples of such storage media are fixed disks (the program may be preloaded), removable disks, memory keys (e.g., USB type), etc.A computer program may be downloaded to a computer device from a storage medium or via a network (e.g., the Internet, a wide area network, and / or a local area network including a transmission cable, an optical fiber, a wireless connection, a network device). One or more network adapters within the computer device receive the computer program from the network and transfer it for storage to one or more storage devices of the computer device. In either case, the techniques according to one embodiment of the present disclosure are useful for implementation in a hardware structure (e.g., an electronic circuit integrated within one or more chips of a semiconductor material, e.g., by a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), etc.) or in a combination of software and hardware appropriately program - set or otherwise configured.

[0174] One embodiment provides a computer device that includes means configured to perform the steps of the method described above. One embodiment provides a computer device that includes circuitry (i.e., any appropriately configured hardware, e.g., by software) for performing each step of the same method. However, the computer device may be of any type (e.g., a central unit of an imaging system, a separate computer, etc.).

[0175] One embodiment provides an imaging system. However, the imaging system may be of any type (e.g., a guided surgical instrument, an endoscope, a laparoscope, etc.).

[0176] In one embodiment, the imaging system includes the computer device described above. However, the computer device may be provided to the imaging system in any manner (e.g., embedded, connected using any wired / wireless connection, etc.).

[0177] In one embodiment, the imaging system includes an illumination unit adapted to apply excitation light to the field of view so as to excite the luminescent substance. However, the illumination unit may be of any type (e.g., based on a laser, LED, UV / halogen / xenon lamp, providing white light or not, etc.).

[0178] In one embodiment, the imaging system comprises an acquisition unit for acquiring a luminescent image and an auxiliary image. However, the acquisition unit may be of any type (e.g., any number and type of lenses, waveguides, mirrors, CCD, ICCD, EMCCD, CMOS, InGaAs, PMT sensors, etc.).

[0179] Generally, similar considerations apply when the computer device and the imaging system have different structures, or are equipped with equivalent components, or have other operating characteristics. In any case, all of its components may be separated into more elements, or two or more components may be integrated into a single element. Further, each component may be replicated to support the execution of corresponding operations in parallel. Further, unless otherwise specified, any interaction between different components generally need not be continuous and may be direct or indirect via one or more mediators.

[0180] One embodiment provides a medical treatment of a patient including the following steps. A region of interest for the medical treatment of the patient (the region of interest includes at least one target body of the medical treatment containing the luminescent substance), and a field of view including one or more foreign objects different from the region of interest are imaged according to the above method, and an output image is output. The medical treatment is carried out with the assistance of the output image. However, the proposed method is applicable to any type of medical treatment (see above).

[0181] In one embodiment, the medical treatment includes administering to the patient a luminescent agent that contains a luminescent substance. However, the luminescent agent may be administered by any method (see above), or this step may be omitted entirely (if the luminescent agent is endogenous).

Claims

1. A method (400) for assisting in the medical treatment of a patient (106), the method (400) being under the control of a computer device (130), a step (406) of obtaining, by the computer device (130), an emission image (205F) of a field of view (103) including a region of interest (109) for the medical treatment of the patient (106) and one or more foreign objects (115-124) that are not of interest for the medical treatment, wherein the region of interest (109) includes at least one target body (112) of the medical treatment containing a luminescent substance, and the emission image (205F) includes a plurality of emission values representing the emission light emitted by the luminescent substance at corresponding emission locations in the field of view (103), the step (406); a step (408) of obtaining, by the computer device (130), an auxiliary image (205R) of the field of view (103), the auxiliary image (205R) including a plurality of auxiliary values representing auxiliary light different from the emission light received from corresponding auxiliary locations in the field of view (103), the step (408); a step (410-424) of dividing, by the computer device (130), according to the content of the auxiliary image (205R), the auxiliary image (205R) into an auxiliary information region (210Ri) representing the region of interest (109) without foreign objects (115-124) and an auxiliary non-information region (210Rn) representing the foreign objects (115-124); a step (428) of identifying, by the computer device (130), an emission information region (210Fi) of the emission image (205F) corresponding to the auxiliary information region (210Ri); a step (432-466) of generating, by the computer device (130), an emission image (205F) by processing the emission image (205F) restricted to the emission information region (210Fi), wherein the processing of the emission image (205F) is based on the emission values of the emission information region (210Fi) to facilitate the identification of the representation of the target body (112) therein, the step (432-466); a step (468-470) of outputting an output image by the computer device (130) based on the processed emission image (220Fi, 225Fi). A method comprising these steps.

2. The method (400) is for assisting in a surgical procedure of the patient (106), The step (468-470) of outputting the output image includes, by the computer device (130), a step (406) of acquiring a light-emitting image (205F) and a step (406) of displaying the output image in substantially real time, the method (400) according to claim 1.

3. The method (400) according to claim 2, wherein the region of interest is the surgical cavity (109) of the patient (106).

4. The method (400) according to any one of claims 1 to 3, wherein at least a part of the foreign objects (115-124) overlaps with the region of interest (109).

5. The method (400) according to any one of claims 1 to 4, wherein the foreign objects (115-124) include one or more medical instruments (115), one or more hands (118), one or more medical tools (121), one or more body parts (124) of the patient (106) that are not of interest for the medical treatment, and / or background material (125).

6. The auxiliary image is a reflected image (205R), The auxiliary light is visible light, The method (400) according to any one of claims 1 to 5, wherein the auxiliary value represents visible light reflected at a corresponding auxiliary location in the field of view (103) illuminated by white light.

7. The method (400) according to any one of claims 1 to 6, wherein the light-emitting substance is a light-emitting agent pre-administered to the patient (106) before performing the method (400).

8. The step (410-424) of dividing the auxiliary image (205R) includes, by the computer device (130), semantically dividing the auxiliary image (205R) into an auxiliary information region (210Ri) corresponding to at least one region-of-interest class of the region of interest (109) and an auxiliary non-information region (210Rn) corresponding to one or more foreign object classes of the foreign objects (115-124), the method (400) according to any one of claims 1 to 7.

9. The step (410-424) of dividing the auxiliary image (205R) includes, by the computer device (130), a step (424) of semantically dividing the auxiliary image (205R) using a neural network, the method (400) according to claim 8.

10. The step (410-424) of dividing the auxiliary image (205R) is Step (418) in which one or more feature maps are determined by the computer device (130) for corresponding features of the auxiliary image (205R), each of the feature maps including a corresponding feature value for an auxiliary location; Step (420-422) in which the computer device (130) semantically segments the auxiliary image (205R) by applying a classification algorithm to the feature values of the feature maps, the method (400) according to claim 8 including this step. **Claim 11** The step (410-424) of segmenting the auxiliary image (205R) The method (400) according to any one of claims 1-10, including step (422) in which, if the auxiliary image (205R) includes one or more cut-off portions completely surrounded by an auxiliary information area (210Ri), the computer device (130) assigns the cut-off portions to the auxiliary information area (210Ri). **Claim 12** Before the step (410-424) of segmenting the auxiliary image (205R), step (410) in which the computer device (130) pre-processes the auxiliary image (205R) by applying histogram equalization to the auxiliary image (205R) in response to the luminance indicator of the auxiliary image (205R) being included between a darkness threshold indicating the feasibility of the step (410-424) of segmenting the auxiliary image (205R) and a brightness threshold higher than the darkness threshold, the method (400) according to any one of claims 1-11 including this step. **Claim 13** The step (410-424) of segmenting the auxiliary image (205R) The method (400) according to any one of claims 1-12, including step (414-424) in which the computer device (130) segments the auxiliary image (205R) according to the content of the light-emitting image (205F). **Claim 14** The step (432-466) of generating the processed light-emitting images (220Fi, 225Fi) The method (400) according to any one of claims 1-13, including step (436-446) in which the computer device (130) performs autoscaling of the light-emitting information area (210Fi) according to the light-emitting values of the light-emitting information area (210Fi). **Claim 15** The step (436-446) of performing autoscaling of the light-emitting information area (210Fi) A step (436) of determining a light emission range of a light emission value of a light emission information area (210Fi) by a computer device (130); A step (438 to 446) of converting a light emission value of a light emission information area (210Fi) according to a mapping function that maps the light emission range to a display range for displaying a light emission image (205F) by a computer device (130), the method (400) according to claim 14.

16. The steps (432 to 466) of generating a processed light emission image (220Fi, 225Fi) include a step (448 to 464) of threshold processing a light emission information area (210Fi) according to a light emission value of the light emission information area (210Fi) by a computer device (130), thereby dividing it into a target segment representing a target body (112) and a non-target segment representing the rest of an area of interest (109) different from the target body (112); The steps (468 to 470) of outputting an output image include a step (470) of outputting an output image by emphasizing a target segment with respect to a non-target segment by a computer device (130), the method (400) according to any one of claims 1 to 13.

17. The steps (448 to 464) of threshold processing a light emission information area (210Fi) include a step (448) of determining a threshold according to a statistical distribution of a light emission value of light emission information (210Fi) by a computer device (130); and a step (450 to 460) of assigning each light emission location of the light emission information area (210Fi) to a target segment or a non-target segment according to a comparison between a corresponding light emission value and the threshold by a computer device (130), the method (400) according to claim 16.

18. The steps (448 to 464) of threshold processing a light emission information area (210Fi) include a step (462) of calculating one or more target statistical parameters of a light emission value of a target segment and / or one or more non-target statistical parameters of a light emission value of a non-target segment by a computer device (130); The step (464) of updating the emission value of the target segment according to the target statistical parameter and / or the non-target statistical parameter by the computer device (130), the method (400) according to claim 16 or 17.

19. The method (400) includes a step (426) of determining, by the computer device (130), corresponding optical values of at least one optical parameter related to the emission light according to the content for the auxiliary location of the auxiliary image (205F) restricted to the auxiliary information area (210Ri). The said steps (432 to 466) of generating the processed emission image (220Fi, 225Fi) include a step (432) of equalizing the emission values of the emission information area (210Fi) according to the optical values by the computer device (130), the method (400) according to any one of claims 1 to 18.

20. The luminescent substance is a fluorescent substance. The emission image is a fluorescence image (205F). The emission value represents the fluorescence emitted by the fluorescent substance at the corresponding emission location irradiated by the excitation light, the method (400) according to any one of claims 1 to 19.

21. A computer program (300) configured to cause the computer device (130) to execute the method (400) according to any one of claims 1 to 20 when the computer program (300) is executed on the computer device (130).

22. A computer program product including a computer-readable storage medium having program instructions collectively stored on a readable storage medium, the program instructions being readable by a computer device and causing the computer device to execute the method according to any one of claims 1 to 20.

23. A computer device (130) including means (300) configured to execute the steps of the method (400) according to any one of claims 1 to 20.

24. The computer device (130) according to claim 23, an illumination unit (133, 139) for applying excitation light suitable for exciting the luminescent substance to the field of view (103), an acquisition unit (145 to 157) for acquiring the emission image (205F) and the auxiliary image (205R), an imaging system (100).

Images