Image processing device and medical treatment device equipped with the same

Abstract

Disclosed herein is an image processing device (1) for processing a fluorescence image obtained from tissue of a subject irradiated with excitation light. The tissue is photosensitized with a fluorescent agent, and the fluorescence image (FI) includes an array of pixels having respective fluorescence signal values. The image processing device is configured to identify transition locations where zero crossings occur in a corrected contrast-to-noise ratio vector (CNR LM (...) ), and to indicate a boundary in the fluorescence image representing the boundary between tumor-containing tissue and healthy tissue based on the identified transition locations. Here, the contrast-to-noise ratio vector (CNR LM (...) ) is defined as follows: Here, FL (...) is a fluorescent signal vector of the scanning trajectory L in the fluorescent image FI, and each value indicates the magnitude of the fluorescent signal in the fluorescent image FI at each position p of the scanning trajectory L. FB is the reference fluorescence signal, which is the average value of the reference fluorescence signal values ​​of the fluorescence image, S is the standard deviation of the reference fluorescence signal values, and c is a predetermined constant. See Figure 1

Description

[Background technology]

[0001] The present invention relates to an image processing device.

[0002] The present invention further relates to a medical treatment device including an imaging device.

[0003] Treatment for most solid cancers consists of radical surgical resection of all tumor tissue. However, distinguishing tumor tissue from normal tissue during surgery remains challenging. Therefore, it is not uncommon to find tumor-positive margins during pathological evaluation 2–5 days after surgery. The literature suggests that tumor-positive margin (TPM) rates range from 10–35%, depending on the tumor type. See, for example, Orosco, RK et al. Positive surgical margins in the 10 most common solid cancers. Sci. Rep. 8, 56–86 (2018). https: / / doi.org / 10.1038 / s41598-018-23403-5. The presence of tumor tissue at or near the margins of resection increases the risk of local recurrence and distant metastasis, resulting in reduced survival. Consequently, TPM necessitates additional treatment, such as reoperation, radiation therapy, or systemic therapy. However, this is associated with increased morbidity and psychological burden for patients. Therefore, accurate observation of tumor tissue during surgery is extremely important, but current optical methods and the surgeon's visual and tactile information alone are insufficient to properly determine tumor boundaries. Therefore, new technologies that enable real-time tumor observation are being researched, with the aim of reducing the number of TPMs and reducing additional treatments and complications.

[0004] One imaging technique that has gained attention is fluorescence molecular imaging (FMI), which allows real-time visualization of tumors both in vivo and immediately after resection (ex vivo). The tissue to be examined is pretreated with a fluorescent agent (FA), either by administering the FA to the patient or by infiltrating the tissue.

[0005] An example of this is published in "Diagnosis of Skin Lesions Using Fluorescence Images" by Suhail M Odeh et al., IMAGE ANALYSIS AND RECOGNITION LECTURE NOTES IN COMPUTER SCIENCE, January 1, 2006 (2006-01-01), LNCS, SPRINGER, Berlin, Germany, pp. 648-659.

[0006] This paper describes a computer-aided diagnosis system for skin lesions. Various parameters and features extracted from fluorescence images are evaluated for cancer diagnosis. The selection of parameters has a significant impact on the cost and accuracy of the automatic classifier. A genetic algorithm (GA) is used to select the parameters using a classifier based on K-nearest neighbors (KNN).

[0007] Fluorescent agents (FAs) can be non-targeted fluorescent dyes, such as indocyanine green (ICG), or targeted fluorescent dyes for imaging tumor tissue or infectious diseases and tracking drug therapy. While FMI studies in the near-infrared (NIR) spectral range (700–900 nm) have shown promising results, complications remain. For example, light scattering and absorption by biological components such as water and blood contribute to attenuation of excitation light, thereby reducing the sensitivity and contrast of fluorescence images. These factors can contribute to false-positive TPMs in fluorescence images, even when no TPMs are present in the patient. Therefore, the resulting fluorescence images (FIs) are not always directly suitable for use as guidance to surgeons and other medical professionals or in medical treatment devices. Summary of the Invention

[0008] According to a first object, an image processing device as claimed in claim 1 is provided, which processes fluorescence images obtained from tissue of a subject to facilitate a specialist or surgical device to perform treatment.

[0009] According to a second object, claim 15 provides a medical treatment device including such an image processing device.

[0010] The image processing device according to the first aspect of claim 1 is configured to process a fluorescence image obtained from tissue of a subject. The tissue is made photosensitive with a fluorescent agent and then illuminated with excitation light. The fluorescence image obtained from the illuminated photosensitive tissue includes an array of pixels having respective fluorescence signal values. The fluorescent agent is used to visualize different types of tissue, such as tumor tissue and non-tumor tissue. In one example, the fluorescent agent is a targeted fluorescent tracer, such as cetuximab-IRDye800CW or Hexbix, for imaging tumor tissue and / or infections and tracking drug therapy, which can be administered to a patient or allowed to infiltrate into tissue. In another example, the fluorescent agent is a non-targeted fluorescent dye, such as indocyanine green (ICG), for imaging tissue perfusion. The excitation light irradiated on the tissue in vivo or ex vivo is typically in the infrared range.

[0011] Currently, there are different types of imaging systems available on the market: those operating in the near-infrared (NIR) spectral range (700-900 nm) and those operating in the shortwave infrared (SWIR) spectral range (1000-1700 nm). The latter type of imaging system is believed to produce higher contrast images and deeper tissue penetration compared to the NIR range due to less autofluorescence and significantly reduced scattering.

[0012] The image processing device provided by the present invention is configured to perform the following operations to determine a representation of a boundary in a fluorescence image between a target region representing a portion of tissue containing a tumor and a reference region outside the target region:

[0013] The image processing device (image processing device) acquires a respective fluorescence signal vector for each scanning trajectory. The series of values ​​of each fluorescence signal vector indicates the magnitude of the fluorescence signal in the fluorescence image at each position of the scanning trajectory. For computational efficiency, the scanning trajectory extends along a straight line, and is therefore sometimes referred to as a scan line. However, other scanning trajectories, such as curved trajectories, can also be considered.

[0014] The image processor then estimates a respective modified contrast-to-noise ratio vector CNR LM (...) for each fluorescence signal vector FL (...). For each scanning trajectory L, the value of the respective contrast-to-noise ratio vector CNR LM (...) is calculated for each position p of the scanning trajectory L as follows:

[0015] where FB is the reference fluorescence signal, which is the average value of the reference fluorescence signal values ​​of the fluorescence image, S is the standard deviation of the reference fluorescence signal values ​​obtained from the background, and c is a predetermined constant.

[0016] In one embodiment of the image processing device, the predetermined constant c used to determine the modified contrast-to-noise ratio is approximately 2, which provides results that closely match the boundaries specified by the pathologist examining the tissue. Alternatively, a value slightly less than 2 can be selected, for example, in the range of 1.5 to 2, in which case the displayed boundaries will extend outside the boundaries specified by the pathologist, providing a safety margin.

[0017] According to another approach, the contrast-to-noise ratio CNR L (p) is determined as follows:

[0018] It is then determined at which point on the line L the contrast-to-noise ratio CNR L (p) exceeds the threshold 2 / c.

[0019] A set of transition locations pt at which LM (…) crosses zero is identified.

[0020] The image processor generates a set of candidate boundary locations based on the set of identified transition locations.

[0021] The image processor generates a representation of the boundary between the target region and the reference region based on the set of candidate boundary locations obtained for each scanning trajectory.

[0022] In one embodiment, the image processor (IPD) is configured to represent the boundary as a linear curve interconnecting the surrounding candidate boundary locations.

[0023] In another embodiment, the image processor is configured to represent the boundary with a quadratic curve that surrounds a primary curve interconnecting the peripheral candidate boundary locations and extends a certain distance outside the primary curve depending on the type of tumor present in the tissue. Guidelines for the distance to be selected, i.e., the tumor-free margin, depend on the tumor type and are shown in Table 1 of Voskuil et al., "Intraoperative Imaging in Pathology-Guided Surgery," Nat Biomed Eng. 6 (2022) 503, https: / / doi.org / 10.1038 / s41551-021-00808-8. In a further elaboration of this embodiment, the image processor is configured to construct the quadratic curve in a manner that avoids intersection with certain anatomical structures.

[0024] In one embodiment of the image processing device, the indication of the magnitude of the fluorescent signal in the fluorescent image at the position of the scanning trajectory is the average of the fluorescent signal values ​​of the pixels in the fluorescent image within a one-dimensional window that includes the position and is oriented transversely to the direction of the scanning trajectory. This operation provides an indication of the magnitude with an improved signal-to-noise ratio, particularly when the scanning trajectory crosses the boundary to be determined in a direction that is substantially transverse to the boundary. In a specific example, the average is determined as a weighted sum of the fluorescent signal values ​​of the pixels within the one-dimensional window according to a Gaussian function that has a maximum value at the position of the scanning trajectory.

[0025] In one embodiment, the image processor is configured to repeat the following steps for each of a plurality of positions along the scan trajectory.

[0026] The image processor tentatively assigns one of a number of positions along the scanning trajectory as an estimate of the boundary of the region of interest.

[0027] The image processor calculates a reference fluorescence signal value as the average of the fluorescence signal values ​​of the fluorescence signal vectors corresponding to positions on the scanning trajectory to the sides of the provisionally assigned position.

[0028] The image processor calculates the standard deviation as the standard deviation of the fluorescence signal values ​​along that side of the scan trajectory.

[0029] In one example, the mean fluorescence signal value is calculated for each position on the scanning trajectory on either side of the provisionally assigned position, and the side with the lowest mean fluorescence signal value is assumed to be within the reference region. Therefore, the lowest mean fluorescence signal value is used as the reference fluorescence signal value, and the standard deviation used is the standard deviation of the fluorescence signal values ​​along the side of the scanning trajectory with the lowest mean fluorescence signal value.

[0030] The image processor identifies the tentatively assigned location as a candidate boundary location if the corrected contrast-to-noise ratio vector zero-crosses at the tentatively assigned location.

[0031] In this embodiment, the plurality of positions typically includes all positions except for both ends of the scanning trajectory, so that the standard deviation can be estimated using at least two signal values. In one example, the image processor is configured to perform low-pass filtering on the modified contrast-to-noise ratio vector, thereby reducing the occurrence of false zero crossings.

[0032] In one embodiment of the image processing device, each scan trajectory includes at least two scan trajectories with different directions. The candidate boundary locations typically indicate locations where the scan trajectories cross the boundary in a direction across the scan trajectory. Therefore, using scan trajectories with different directions can provide a more complete set of candidate boundary locations. For example, the scan trajectories include a first set of scan trajectories in horizontal directions and a second set of scan trajectories in vertical directions. This embodiment is highly computationally efficient. At the expense of increased computational effort, additional candidate boundary locations can be calculated using more scan trajectory directions. For example, a scan trajectory set can be used for each direction that is a multiple of n degrees (e.g., n is 10).

[0033] As mentioned above, the image processing device can be applied to both in vivo and in vitro analysis. When the image processing device is used for in vitro analysis,

[0034] The fluorescence image is obtained from a tissue sample taken from a subject and then placed on a background, typically a dark background. In one embodiment, the image processor is configured to identify regions in the fluorescence image representing the tissue sample as tissue regions and regions in the fluorescence image representing the background as background regions. In one example of this embodiment, the image processor is further configured to determine a first maximum intensity in the region of interest in the fluorescence image, determine a second maximum intensity along the scanning trajectory, and skip further processing steps for the scanning trajectory if the second maximum intensity is less than a predetermined percentage of the first maximum intensity.

[0035] In one embodiment, the image processing device is alternatively or additionally configured to exclude a candidate boundary location from further processing if the fluorescence intensity indicator is below a threshold value. The indicator can be the intensity value of the fluorescence image at the location of the candidate boundary location to be determined. However, to reduce noise sensitivity in this candidate boundary location filtering process, the indicator can be the maximum fluorescence intensity value within a window centered on the candidate boundary location. This is the value indicated by the "movmax" function, and the window size can be selected from a range of 2 to 10 pixels, e.g., 5 pixels.

[0036] In one embodiment, the imaging device includes a camera for acquiring a fluorescence image from tissue of a subject, and further includes a display device for displaying the tissue and one or more indicia indicating its boundary, e.g., as a linear curve and / or a quadratic curve surrounding the linear curve. This embodiment is particularly suitable for use during treatment to guide a medical professional in the process of tumor ablation or tumor irradiation, where the patient undergoing treatment has been administered a fluorescent agent.

[0037] According to a second aspect of the present application, there is provided a therapeutic device for treating a subject to which a fluorescent agent has been administered prior to treatment. The therapeutic device includes an excitation light source that irradiates tumor-containing tissue within the subject with excitation light, a camera that acquires a fluorescent image from the irradiated tissue, the image processing device described above, and a therapeutic device that performs treatment to remove or shrink the tumor from the subject according to the boundary indicated by the image processing device. In one example, the therapeutic device is a surgical robot that excises the tumor from the tissue. In another example, the therapeutic device is a therapeutic irradiation device that irradiates the tumor with radiation. In yet another example, the therapeutic device is a therapeutic irradiation device that activates a therapeutic substance within a range specified by the boundary. [Brief explanation of the drawings]

[0038] These and other aspects of the present disclosure will be described in more detail with reference to the drawings. FIG. 1 shows a schematic representation of an embodiment of an image processing device according to a first aspect. FIG. 2 illustrates an exemplary module in another embodiment of the present invention. FIG. 3 shows further exemplary modules. FIG. 4 shows some of the modules of FIG. 3 in more detail. FIG. 5 shows a further embodiment of an image processing device according to the first aspect of the invention. FIG. 6 shows an embodiment of a medical treatment device according to the second aspect. FIG. 7A shows a fluorescent image taken from tissue containing a tumor. FIG. 7B shows the result of the first transformation step applied to the fluorescence image. FIG. 7C shows the result of a second transformation step applied to the result shown in FIG. 7B. FIG. 8 shows the first data processing operation. 9A and 9B show the results obtained in the second data processing run. 10A, 10B and 10C show an embodiment of a third data processing operation. 11A and 11B show the results of a variation of the fourth data processing operation. 12A and 12B show the results of a variation of the fifth data processing operation. 13A and 13B show the results of a variation of the fifth data processing operation. FIG. 14 shows the results of the sixth data processing operation. FIG. 15 shows the results of the seventh data processing operation. FIG. 16 shows the results of the eighth data processing operation. FIG. 17 shows a representation of the boundary superimposed on the fluorescent image of FIG. 7A. 18A-18F show first exemplary results obtained from fluorescence images obtained in vivo from patient tissue. 19A-19F show a second exemplary result obtained from fluorescence images taken of ex vivo tissue after it has been completely excised from the patient. 20A-20F show a third exemplary result obtained from fluorescence images taken ex vivo from excised tissue sections. 21A-21F show a fourth exemplary result obtained from fluorescence images acquired in vivo from tissue of another patient. 22A-22F show a fifth exemplary result obtained from fluorescence images acquired ex vivo from intact tissue excised from a patient. 23A-23F show a sixth exemplary result obtained from fluorescence images taken ex vivo from excised tissue sections. Figures 24A-24F show seventh results obtained from fluorescent images taken from tissue sections. DETAILED DESCRIPTION OF THE INVENTION

[0039] 1 shows a schematic diagram of an image processing device 1 configured to process a fluorescence image FI of tissue of a subject. The tissue is sensitized with a fluorescent agent (e.g., a contrast agent) and irradiated with excitation radiation. The resulting fluorescence image FI includes an array of pixels having respective fluorescence signal values. The image processing device 1 is configured to determine a boundary in the fluorescence image between a target region TR representing a portion of the tissue containing a tumor and a reference region RR outside the target region.

[0040] The image processing device 1 includes a signal vector acquisition module 10 configured to acquire a respective fluorescence signal vector FL (...) for each scanning trajectory L, as indicated by a scanning trajectory selection module 11. Each value of the fluorescence signal vector FL (...) indicates the magnitude of the fluorescence signal in the fluorescence image FI at each position p of the scanning trajectory L.

[0041] The contrast-to-noise ratio calculation module 12 calculates a corrected contrast-to-noise ratio vector CNR LM (...) for each fluorescence signal vector FL (...). The corresponding value contained in the contrast-to-noise ratio vector CNR LM (...) is calculated from the fluorescence signal vector FL (...) at each position p of the scanning trajectory L as follows:

[0042] The statistical module 13 provides a reference fluorescence signal FB, which is the mean value and standard deviation S of the reference fluorescence signal values ​​in the fluorescence image, where c is a predetermined constant, for example c=2.

[0043] The zero-crossing detection module 14 respectively identifies a set of transition positions {pt} along each scan trajectory, which are the positions where the modified contrast-to-noise ratio vector CNR LM (...) crosses zero.

[0044] The scan trajectory selection module 11 selects multiple scan trajectories that are different from each other and determines a respective set of transition positions for each selected scan trajectory.

[0045] The boundary display module 15 then displays the boundary between the target region TR and the reference region RR based on the set of transition positions obtained for each scanning trajectory.

[0046] In the illustrated embodiment, the image processing device 1 further includes a display module 16, which generates a modified fluorescence image FI by superimposing the boundary B displayed by the boundary display module 15 on the original fluorescence image FI. If the original fluorescence image FI was obtained in vivo, this image can assist a surgeon in resecting the tumor or irradiating it with therapeutic radiation.

[0047] Next, the scan trajectory selection module 11 selects a scan trajectory L from the set of scan trajectories and instructs the signal vector search module 10 to obtain a fluorescence signal vector. Each fluorescence signal vector value of the fluorescence signal vector indicates the magnitude of the fluorescence signal at each position p of the scan trajectory L in the fluorescence image FI. Each position p has coordinates xp, yp in the fluorescence image FI. In one example, the fluorescence signal vector value corresponds to the magnitude of the fluorescence signal at each position p of the scan trajectory L in the fluorescence image FI. In another example, an index FL(p) indicating the magnitude of the fluorescence signal in the fluorescence image FI at the position of the scan trajectory L is the average value of the fluorescence signal values ​​of pixels within a one-dimensional window including the position of coordinates xp, yp in the fluorescence image FI and oriented laterally with respect to the direction of the scan trajectory.

[0048] Accordingly, here

[0049] Here, the position xq =0, yq =0 coincides with the position xp, yp. For example, the weighting factors are equal to each other, i.e. w_q=1 / (2n+1).

[0050] In a preferred embodiment, the mean value is a weighted sum of the fluorescence signal values ​​of the pixels within a one-dimensional window according to a Gaussian or similar weighting function that has a maximum at position p of the scan trajectory L.

[0051] When the image processing device 1 is used in an intraoperative mode, a fluorescence image FI is obtained from a tissue sample taken from a subject and placed on a background. In one embodiment, the image processing device 1 includes a foreground identification module 17 (shown in dashed lines), which is configured to identify regions in the fluorescence image representing the tissue sample as tissue regions and to identify regions of the background in the fluorescence image FI as non-tissue regions. In one example of this embodiment, the foreground identification module 17 (shown in dashed lines) is further configured to determine a first maximum intensity in the tissue region of the fluorescence image FI, determine a second maximum intensity along the scanning trajectory, and skip further processing steps for the scanning trajectory if the second maximum intensity is less than a predetermined percentage of the first maximum intensity.

[0052] Assume that B and the standard deviation S are given values.

[0053] FIG. 2 shows an alternative embodiment of the statistics module 13 that dynamically determines these parameters FB and S 1 as a function of the fluorescence signal vector data FL.

[0054] The statistical module 13 includes a vector division unit 130 that divides the fluorescence signal vector FL(...) into a first portion FLO and a second portion FLI. The first portion FLO contains fluorescence signal vector FL(...) data corresponding to a first side of the scanning trajectory relative to the position ps of the scanning trajectory, which is provisionally assigned by the position assignment unit 131. The second portion FLI contains fluorescence signal vector FL(...) data corresponding to the opposite side of the scanning trajectory relative to the position ps. Each portion may or may not contain the fluorescence signal vector FL value for the position ps. The first average value calculation unit 132 calculates the average value FB0 of the values ​​of the first portion FLO(...) of the fluorescence signal vector FL(...). Similarly, the second average value calculation unit 133 calculates the average value FB1 of the values ​​of the second portion FL1(...) of the fluorescence signal vector FL(...). Alternatively, a single unit may continuously calculate these values ​​FB0 and FB1. The determination unit 134 determines whether the value FB0 exceeds the value FB1. If so, it determines that FB = FB1. Otherwise, it determines that FB = FB0. The determination unit 134 also outputs a control signal S0 / 1 to the data selector 135. If FB = FB1, the determination unit 134 controls the data selector 135 to select the second portion F L1 (...) of the fluorescence signal vector FL (...) as input to the standard deviation calculation unit 136, which calculates the standard deviation S from the fluorescence signal data of the second portion F L1 (...) of the fluorescence signal vector FL (...). In the second case, if FB = FB0, the data selector 135 is controlled to select the first portion F L0 (...) of the fluorescence signal vector FL (...) as input. In one embodiment, the position assignment unit 131 assigns successive positions ps of the scanning trajectory, and for each tentatively assigned position, determines the corresponding parameters FB and S and uses them to calculate the respective modified contrast-to-noise ratios. Furthermore, the zero-crossing detection module 14 specifically selects tentatively assigned locations as candidate boundary locations if the modified contrast-to-noise ratio vector (CNR LM (...) ) has zero crossings at those tentatively assigned locations.

[0055] An embodiment of the boundary display module 15 is shown in more detail in FIG.

[0056] The illustrated boundary indication module 15 includes a candidate boundary location filter 150 configured to exclude a candidate boundary location from further processing if the indication of the fluorescence intensity is below a threshold value. In one example, the indication is the maximum value of the fluorescence intensity within a window centered on the candidate boundary location to be determined. In one example, the maximum value is obtained from an auxiliary image FIa, which is derived once from the original fluorescence image and stored for future reference. The auxiliary image FIa is derived, for example, in a two-pass procedure, where in the first pass, for each pixel (x,y), a value FIh(x,y) is determined as follows:

[0057] And on the second pass

[0058] In one example, the threshold is selected as a percentage of the maximum value in the entire fluorescence image FI. The percentage can be selected between 10 and 80% depending on the tissue being imaged, but is more preferably between 40 and 60%. For samples with smooth surfaces, such as a slice of bread, 40% is an appropriate choice. For samples with greater surface roughness, the percentage should be higher. The window size 2w+1 is selected between 2 and 10, but a value of 5, corresponding to w = 2, was found to be optimal.

[0059] The embodiment of the boundary display module 15 shown in FIG. 3 further includes a candidate boundary position matching unit 151 that matches candidate boundary positions obtained from each pair of overlapping but oppositely oriented scanning trajectories. The candidate boundary position matching unit 151 verifies whether a candidate boundary position obtained from a first scanning trajectory of a pair has a corresponding candidate boundary position in a second scanning trajectory of the pair. Candidate boundary positions are considered to correspond if their positions in the fluorescence image do not exceed a threshold distance (e.g., one pixel or two pixels). Furthermore, the candidate boundary positions may be required to be of opposite polarity. That is, if one candidate boundary position indicates a boundary crossing in the direction from the reference region to the target region, the other candidate boundary position must indicate a boundary crossing in the direction from the target region to the reference region.

[0060] In the illustrated embodiment, the boundary display module 15 includes a further candidate boundary position matching unit 152 that matches pairs of adjacent but non-coincident candidate boundary positions (i.e., pairs having a relative angle other than 0 or 180 degrees) obtained from different scanning trajectories. The further candidate boundary position matching unit 152 identifies pairs of first and second candidate boundary positions obtained from a first scanning trajectory whose Euclidean distance is less than a predetermined maximum value (e.g., 3 pixels), and replaces these pairs of adjacent candidate boundary positions with a single replacement candidate boundary position located in the middle of these adjacent candidate boundary positions. Note that instead of the Euclidean distance measure, a different distance measure can be adopted, such as the L1 measure, which calculates the distance as the sum of the distances along the main directions of the image, or the L∞ measure, which defines the distance as the maximum distance in the main directions.

[0061] In the illustrated embodiment, the boundary display module also includes a candidate boundary position integrator 153. The candidate boundary position integrator 153 receives the set of remaining candidate boundary positions {pt}b from the further candidate boundary position matcher 152 and integrates adjacent candidate boundary positions among the remaining candidate boundary positions. As indicated by the backward arrow "N," the candidate boundary position integrator 153 can perform this operation multiple times. The determiner 154 determines whether the iterative process should be stopped with "Y" or "N." In one example, the iterative process is performed a predetermined number of times, for example, 3 to 5 times. In another example, the determiner 154 determines whether a data-dependent stopping criterion is met, for example, whether the number of remaining candidate boundary positions has decreased below a threshold. The threshold may be predetermined or may depend on other information. For example, the threshold is the square root of the total number of pixels in the convex hull containing all remaining candidate boundary positions multiplied by a constant.

[0062] The boundary constructor 155 receives the set of candidate boundary positions {pt}n selected based on the set of transition positions from the candidate boundary position merger 153, and constructs an indication of the boundary B.

[0063] In one example, the boundary constructor 155 implements the MatLab function boundary(x,y,s), where the value s can be chosen in the range 0.1 to 1, typically in the range 0.5 to 1. Best results have been obtained when chosen in the range 0.8 to 1, although the default value of s = 0.5 also gives good results.

[0064] 3, in some examples, boundary display module 15 is configured to omit one or more of the above-specified processes. For example, boundary constructor 155 constructs a boundary directly from the filtered set of candidate boundary locations {pt} a obtained from candidate boundary location filter unit 150. In another example, boundary constructor 155 constructs a boundary from the set of candidate boundary locations {pt} c obtained from additional candidate boundary location matcher 152.

[0065] In one example, the boundary construction section 155 is configured to show boundary B as a major curve that interconnects the peripheral transition locations.

[0066] 4, the boundary constructor 155 is configured to construct the boundary as a quadratic curve B' that surrounds a linear curve that interconnects the peripheral transition positions and extends a predetermined distance outside the linear curve depending on the type of tumor present in the tissue. To this end, the boundary constructor 155 receives a control signal C1 that specifies the predetermined distance from the distance control unit 156. In response to this, the boundary constructor 155 displays the boundary as a quadratic curve B' that extends outside the linear curve such that the distance between each point on the quadratic curve and the closest corresponding point on the linear curve is equal to the predetermined distance.

[0067] In a variation of the embodiment of Figure 4, boundary constructor 155 is further configured to construct a quadratic curve B' to avoid intersection with certain anatomical structures. To this end, this variation also includes a vital tissue protection unit 157, as shown by the dashed line in Figure 4. Vital tissue protection unit 157 has information regarding the location of vital tissue, such as vital organs. Based on this information, boundary constructor 155 constructs quadratic curve B' to extend outward from primary curve B by a predetermined distance by default, but locally to extend a shorter distance to avoid intersection with vital organs.

[0068] FIG. 5 shows an embodiment of an image processing device 1 particularly suitable for use as a tool during treatment. In this case, it is assumed that a fluorescent agent has been administered to the patient. In this embodiment, the image processing device 1 is configured, for example, as shown in FIG. 5. FIG. 1 or a variation thereof further includes a camera 2 for acquiring a fluorescence image FI from the patient's tissue to be treated and a display device 3 for displaying an indication B of the tissue and boundary shown in the corrected fluorescence image FI. In a variation, the image processing device 1 displays a quadratic curve B' superimposed on the fluorescence image, as described above, optionally in combination with an indication by the linear curve B. In the illustrated example, the image processing device 1 is further configured to issue a control signal C4 for controlling an excitation light source 4, e.g., a near-infrared (NIR) or short-wave infrared (SWIR) light source. This allows the image processing device 1 to control the wavelength and intensity of the excitation light source 4 to optimize the contrast of the fluorescence image FI. Alternatively, a different excitation light source 4 with predetermined settings known to produce good results may be used.

[0069] Figure 6 shows an embodiment of a medical treatment device 100 suitable for automatically administering a medical treatment to a patient to treat a tumor present in tissue. As in Figure 5, it is assumed that a fluorescent agent has been administered to the treatment target. In addition to an image processing device 1 such as that of Figure 1 or a variation thereof, the medical treatment device 100 includes a camera 2 for acquiring a fluorescent image FI from tissue containing the tumor, and a medical treatment device 5 for performing treatment by ablating or irradiating the tumor according to a specified boundary, or by activating a therapeutic substance within a range specified by the specified boundary.

[0070] Similar to the embodiment of FIG. 5, the image processing device 1 may further be configured to issue a control signal C4 for controlling the excitation light source 4, e.g., a near-infrared (NIR) or short-wave infrared (SWIR) light source. Even if the medical device 100 is suitable for automatically performing a medical procedure, it may be desirable to monitor its operation. To that end, a display device 3 may be provided for displaying the modified fluorescence image FI, as shown by the dashed line. Furthermore, the medical device 5 may provide an information signal I5 to the display device 3 indicating the progress of the treatment. The medical device 5 may also be connected to a user interface 6 that allows a medical professional to fully or partially control the operation of the medical device 5 via the control signal C5.

[0071] Example 1

[0072] As an example, a fluorescence image FI taken from tumor-containing tissue on a black background is shown. In this example, the width Nx of the fluorescence image is 400 pixels, the height Ny is 600 or 700 pixels, and the resolution is, for example, about 100 microns per pixel, e.g., about 85 microns.

[0073] In the example shown, the tissue from which the fluorescent image was obtained is a section of a tissue volume excised from a subject. The tumor in this case was penile cancer, and the subject had been treated prior to receiving cetuximab-IRDye800CW as the fluorescent agent.

[0074] Alternatively, fluorescent images are acquired from the excised tissue volume, i.e., the excision specimen itself.

[0075] In the subsequent processing, the image processing device 1 identifies a first region representing the tissue sample in the fluorescence image FI as a tissue region, and identifies a background region in the fluorescence image FI as a background region BA. In one example, these processes are performed by the foreground identification module 17.

[0076] In a first exemplary operation, the foreground identification module 17 converts the fluorescence image FI into a binary image, as shown in FIG. 7B. Pixels in the fluorescence image whose intensity exceeds a threshold have a first binary value, represented as white pixels in FIG. 7B, while the remaining pixels have a second binary value, represented as black pixels in the figure. The threshold value selected for this purpose depends on the selection of the input device providing the fluorescence image FI. For the PEARL imaging system, the threshold value can be selected in the range of 0.001 to 0.01. For example, the threshold value is 0.005. For the SurgVision imaging system, the threshold value can be selected in the range of 500 to 2000. For example, the threshold value is approximately 800.

[0077] The region formed by pixels with the first binary value is designated as a possible foreground region, and the region formed by pixels with the second binary value is designated as a possible background region.

[0078] Next, the foreground identification module 17 performs a dilation operation to expand the candidate foreground regions. Typically, a dilation of one pixel is sufficient, avoiding the possibility of losing image data at the edges of the tissue. As shown in Figure 7B, due to the effects of noise, in addition to one large candidate foreground region, there are multiple smaller candidate foreground regions. To mitigate this phenomenon, the foreground identification module 17 performs an additional operation to select the largest candidate foreground region TA as the foreground region representing tissue. This results in the result shown in Figure 7C. The resulting binary image in Figure 7C is used as a mask in the fluorescence image FI to indicate the boundary between the foreground region representing tissue and the remaining background region.

[0079] In one example, the foreground identification module 17 also determines the maximum intensity Imax,fg in the foreground region (denoted as TA in FIG. 7C) of the fluorescence image FI. It is also possible to determine the maximum intensity over the entire region of the fluorescence image FI, but this is more susceptible to noise since isolated pixels may also have high intensities in the background region.

[0080] As shown in FIG. 8, the scan line selection module 11 constructs at least one scanning trajectory L that penetrates the foreground region TA (see FIG. 7C) and is delimited by a boundary with the background region. In the illustrated example, the scanning trajectory is a line extending horizontally between a first boundary with the background at position x0 and a second boundary with the background at position x1. The signal vector acquisition module 10 acquires a fluorescence signal vector FL(...) based on the scanning trajectory L. Each value of the fluorescence signal vector FL(...) indicates the magnitude of the fluorescence signal in the fluorescence image FI at each position p of the scanning trajectory L. The constructed scanning trajectory may represent a strip with a width greater than one pixel. In this case, each value of the fluorescence signal vector FL(...) is a weighted sum of the intensities of pixels within the width of the strip. That is, if the line extends in the first direction x, the strip has a width in the y direction, and a weighted sum of pixels with the same x coordinate within the width of the strip is calculated. The weighted sum can be a simple average, in which each pixel has the same weighting coefficient, or a Gaussian weighting function. If the foreground region is concave, multiple trajectories may be constructed on the same line, each with endpoints corresponding to two points on the boundary. We experimented with varying the strip width from 3 to 21 pixels. The best results were obtained with a strip width of 11 pixels, but other values ​​within this range proved to be suitable.

[0081] In one example, the scan line selection module 11 determines the maximum value (denoted as Imax,ln) of the fluorescence signal vector FL (...) for each scan trajectory L. In this example, it also determines whether the maximum intensity Imax,ln is equal to or greater than a predetermined percentage (e.g., approximately 10%) of the maximum intensity Imax,fi of the foreground region of the entire fluorescence image. The predetermined percentage was selected from 0, 1, 10, 33, and 50%, and experiments were conducted with various values. The best results were obtained when the predetermined percentage was 33.

[0082] If the maximum intensity Imax,ln among the values ​​of L (…) is determined to be less than a predetermined percentage of the maximum intensity Imax,f, the scan trajectory L is not selected for further processing. This deselection is based on the assumption that the absence of high-intensity pixels along the scan trajectory indicates the absence of tumor tissue in the path through the tissue corresponding to the scan trajectory. The predetermined percentage can be selected in the range of 10% to 50%. Experiments have shown that 40% is optimal.

[0083] If the maximum intensity Imax,ln among the values ​​of L(...) is determined to be equal to or greater than a predetermined percentage of the maximum intensity Imax,fi, the contrast-to-noise ratio calculation module 12 calculates a corrected contrast-to-noise ratio vector CNRLM(...) of the fluorescence signal vector FL(...). Based on the corrected contrast-to-noise ratio vector CNRLM(...), one or more positions where the scanning trajectory passes through the boundary between the subarea containing healthy tissue and the subarea containing tumor tissue are estimated. The position where the scanning trajectory crosses the boundary from the subarea containing healthy tissue to the subarea containing tumor tissue is characterized by a zero-crossing of the corrected contrast-to-noise ratio vector CNRLM(...) in the direction from negative to positive polarity, and the position where the scanning trajectory crosses the boundary from the subarea containing tumor tissue to the subarea containing healthy tissue is characterized by a zero-crossing of the corrected contrast-to-noise ratio vector CNRLM(...) in the direction from positive to negative polarity.

[0084] In one example, the contrast-to-noise ratio calculation module 12 calculates the contrast-to-noise ratio vector CNR LM (...) based on predetermined values ​​of the background intensity FB and the standard deviation S. This implementation has the advantage of relatively low computational complexity. An alternative implementation that can more accurately determine the boundary includes repeating the following procedure for each pixel xs on the scanning trajectory, excluding both ends of the scanning trajectory. For simplicity, it is assumed here that the scanning trajectory extends in the x direction, and the boundaries between the foreground and background are x0 and x1, respectively. However, this description is equally applicable to any direction.

[0085] In this improved procedure, the selected pixel xs coincides with the boundary, and it is estimated that all pixels on the scanning trajectory on the first side of the selected pixel xs represent tumor tissue, and all pixels on the scanning trajectory on the second side of the selected pixel (opposite the first side) represent healthy tissue.

[0086] Based on this assumption, the vector division unit 130 divides the fluorescence signal vector FL(...) into a first portion F L0 and a second portion F L1. The first portion F L0 represents the portion of the scanning trajectory between the first endpoint (here, x-coordinate x0) and the provisionally assigned boundary position xs. The second portion F L1 represents the portion of the scanning trajectory between the provisionally assigned boundary position xs and the second endpoint (here, x-coordinate x1).

[0087] Based on this assumption, the first average value calculation section 132 calculates the average value FB0 of the values ​​in the first portion F L0 of the fluorescence signal vector FL (...). Similarly, the second average value calculation section 133 calculates the average value FB1 of the values ​​in the second portion F L1 of the fluorescence signal vector FL (...).

[0088] Thus, if ps (here xs) is a tentatively selected position on the scan trajectory, and p0 (here x0) and p1 (here x1) are the positions of the end pixels, then the average intensity on each side of that position ps is determined as follows:

[0089] The tumor is considered to be represented by the pixel on the posterior side of the scan trajectory with the highest mean intensity, while the pixel on the posterior side of the scan trajectory with the lowest mean intensity is considered to represent the background, as determined by decision section 134.

[0090] Based on this estimation, the mean value FB and standard deviation S of the pixel values ​​of the pixels on the scanning trajectory on the second side of the selected pixel are calculated.

[0091] Therefore, if FB0 > FB1, it is estimated that the tumor is present in the range p0 to ps, and the value of FB1 represents the estimated mean background intensity level FB. In this case, the decision unit 134 uses the selection signal S0 / 1 to cause the data selector 135 to select the second part FL1 of the fluorescence signal vector FL(...) as input to the standard deviation calculation unit 136. Therefore, in this case, the standard deviation S is estimated as follows:

[0092] Alternatively, if FB0 - FB1, it is estimated that the tumor is present in the range p-p1, and the value of FB0 represents the estimated mean background intensity level FB. In this case, the decision unit 134 uses the selection signal S0 / 1 to cause the data selector 135 to select the first part FLO of the fluorescence signal vector FL(...) as input to the standard deviation calculation unit 136. Therefore, in this case, the standard deviation S is estimated as follows:

[0093] Based on the values ​​of the estimated mean background intensity FB and standard deviation S for the provisionally assigned position ps, the contrast-to-noise ratio calculation module 12 calculates the corrected contrast-to-noise ratio CNR LM (p) as follows:

[0094] Thus, for each estimated ps a vector CNR LM (…) is obtained, resulting in a matrix in which each row represents the value of CNR LM (…) calculated for a particular provisionally assigned position ps.

[0095] FIG. 9A shows, as an example, the values ​​of the fluorescence signal vector FL(p) along the scanning trajectory and the average values ​​FB0 and FB1 calculated for the first and second portions of the fluorescence signal vector FL(p) based on the provisionally selected position p. In this example, the value of FB1 is smaller than the value of FB0, and it is estimated that the portion p0-p is within the image region representing the tumor, and the portion p-p is within the region representing healthy tissue. Therefore, the value of FB1 is selected as the background value FB, and the standard deviation S is calculated from the values ​​of the fluorescence signal vector FL(p) in the range p-p.

[0096] This allows the calculation of the corrected contrast-to-noise ratio vector CNRLM(…) from the values ​​of the fluorescence signal vector FL(p), as shown in Figure 9B.

[0097] With this input, the zero-crossing detection module 14 determines at which values ​​of the tentatively assigned position p the modified contrast-to-noise ratio vector CNR LM (...) obtained for that position p actually has a zero crossing. The corresponding positions of the scanning trajectory in the fluorescence image FI are considered to represent candidate boundary positions. In the example shown in Figures 9A and 9B, it can be seen that the modified contrast-to-noise ratio vector CNR LM (...) obtained for the tentatively assigned position p does not have a zero crossing at that position p. Therefore, this position p is not considered a candidate boundary position.

[0098] A low-pass filter can be applied to LM(...) before detecting the zero crossings to remove false zero crossings due to noise. In one example, the normalized passband frequency of the low-pass filter is less than 0.5, typically less than 0.1. Optimal results have been obtained with a normalized passband frequency of 0.045.

[0099] Therefore, it is sufficient to determine the presence of a zero crossing at position p of the modified contrast-to-noise ratio vector CNR LM (...) that coincides with the provisionally assigned position p. That is, it is sufficient to calculate the modified contrast-to-noise ratio vector CNR LM (...) only in the vicinity of position p, optionally taking into account a low-pass filter window. However, it may be advantageous to completely calculate each modified contrast-to-noise ratio vector CNR LM (...), so that subsequent processing can be easily parallelized using a vector processor.

[0100] Note that the output of the zero-crossing detection module 14 may include a set of multiple candidate boundary locations {pt}. The set of candidate boundary locations may include a first candidate boundary location where the scanning trajectory crosses the boundary in a direction from a region representing healthy tissue in the image to a region representing tumor-containing tissue, and a second candidate boundary location where the scanning trajectory crosses the boundary in a direction from a region representing tumor-containing tissue to a region representing healthy tissue in the image. The set may be empty if the scanning trajectory does not enter a region representing non-healthy tissue. In some cases, the boundary between the image region representing tumor-containing tissue and the image region representing healthy tissue may be concave. In such cases, the zero-crossing detection module 14 may identify multiple pairs of candidate boundary locations for the scanning trajectory.

[0101] As shown in FIG. 10A, during operation, the image processing device 1 performs the above-described procedure for multiple scanning trajectories L1, ...Ln. The distance between the scanning trajectories is typically correlated with the strip width, as described above. If the strip width is wide, the distance between the scanning trajectories may be greater. For example, the distance between the scanning trajectories may be selected to be equal to the strip width.

[0102] In some embodiments, all scan trajectories are parallel, but this complicates the detection of tissue boundaries that run in the same direction as the scan trajectories.

[0103] This can be avoided by performing the procedure using at least two sets of scan trajectories with different orientations. In one example, subsequent sets of scan trajectories differ in orientation by 45 degrees. Figures 10A, 10B, and 10C show three such sets of scan trajectories. In another example, the procedure is performed using two sets of scan trajectories with mutually orthogonal orientations. In this case, the choice of the x and y directions corresponds to the orientation of the major axis in the fluorescence image, which is preferable for computational simplicity. Alternatively, the procedure can be performed using more sets of scan trajectories, for example, angles 0, 10, 20, … degrees relative to the major axis of the fluorescence image. It is also possible to perform the procedure using an even larger set of scan angles, for example, scan angles differing by 1 degree from each other. However, this is not recommended because it involves additional computational load and does not substantially contribute to performance in terms of accuracy. In practice, a difference of 10 degrees has been found to be optimal.

[0104] The total number of pixels in the scanning trajectory must cover at least a predetermined percentage of the total number of pixels in the foreground region.

[0105] If the scanning trajectory represents a strip of pixels, the predetermined percentage will be relatively small.

[0106] The zero-crossing detection module 14 repeats the zero-crossing estimation procedure for all scan trajectories, generating multiple candidate boundary locations for the boundary between healthy and unhealthy tissue. Figure 11A shows the candidate boundary locations superimposed on the original fluorescence image FI obtained using 36 scan trajectories with angles differing by 10 degrees. Figure 11B shows another result obtained using four scan trajectories with angles differing by 90 degrees.

[0107] The boundary display module 15 generates a representation of the boundary between regions representing tumor tissue and regions of healthy tissue in the image based on the set of candidate boundary locations obtained from the zero-crossing detection module 14 .

[0108] In one example, the boundary display module 15 compares the candidate boundary locations of each scan trajectory with the candidate boundary locations of each overlapping scan trajectory extending in the opposite direction. In this example, the boundary display module 15 determines which candidate boundary locations of the first scan trajectory match the candidate boundary locations of the overlapping, opposite-direction scan trajectory. A pair of candidate boundary locations is determined to match if they match or are close together, i.e., if their Euclidean distance is less than or equal to a distance threshold (e.g., a value selected from the range of 1 to 10). Best results have been obtained with values ​​in the range of 2 to 5. When overlapping, opposite-direction scan trajectories extend along the primary coordinate axes x and y of the image, the Euclidean distance measure is reduced to the absolute difference between the coordinate values ​​of the compared candidate boundary locations. In one embodiment, the boundary display module 15 eliminates candidate boundary locations that do not have a matching candidate boundary location in the overlapping, opposite-direction scan trajectory. Exemplary results of this operation are shown in Figures 12A and 12B. Figure 12A shows the results of applying the pruning procedure to the set of candidate boundary locations obtained from 36 pairs of scan trajectories differing in angle by 10 degrees, as shown in Figure 11A. Figure 12B shows the results of applying the pruning procedure to the set of candidate boundary locations obtained from 4 pairs of scan trajectories differing in angle by 90 degrees, as shown in Figure 11B.

[0109] A further implementation of boundary display module 15 in this example also performs a merging operation: if a pair of candidate boundary locations obtained from overlapping, opposite scanning trajectories do not match but are close enough to satisfy a distance threshold, the pair of candidate boundary locations is replaced with a single candidate boundary location centered between the two original candidate boundary locations. The result of this operation is shown in Figure 13A, where the merging procedure has been applied to the set of remaining candidate boundary locations as shown in Figure 12A. Similarly, Figure 13B shows the result of applying the merging procedure to the set of remaining candidate boundary locations as shown in Figure 12B.

[0110] In one variation, a confidence index is associated with the original candidate boundary location. This confidence index is calculated, for example, based on the magnitude of the gradient of the fluorescence signal vector FL(...) of the scan trajectory from which the original candidate boundary location was obtained. A larger magnitude of the gradient indicates a higher confidence in the location of the boundary location. For example, a single candidate boundary location is located close to the original candidate boundary location with the highest confidence index.

[0111] In one embodiment, boundary display module 15 performs a process to filter out isolated candidate boundary locations. In this embodiment, boundary display module 15 identifies a candidate boundary location as an isolated candidate boundary location if there are no other candidate boundary locations within a predetermined distance from the candidate boundary location in the complete set of candidate boundary locations. This predetermined distance may be the same as the predetermined distance used in the process described above to determine whether a candidate boundary location obtained from one scan trajectory matches a candidate boundary location from an overlapping scan trajectory in the opposite direction. An example result of this filtering process is shown in FIG. 14. In this example, the filtering process is applied to the remaining set of candidate boundary locations, as shown in FIG. 13A.

[0112] In one embodiment, the boundary display module 15 performs a process to merge pairs of candidate boundary locations that are within a threshold distance of each other into a single candidate boundary location. Unlike the previously described merging procedure for merging pairs of candidate boundary locations obtained from overlapping, oppositely oriented scan trajectories, this merging procedure is applied to any pair of candidate boundary locations within the set of available candidate boundary locations that are within a threshold distance specified for this process. The threshold distance specified for this process may be selected from a range of 2 to 5. FIG. 15 illustrates an exemplary result of applying the merging process to the set of candidate boundary locations shown in FIG. 14. In this example, the merging process is applied once; however, alternatively, the process may be repeated a predetermined number of times or until a stopping criterion (e.g., the number of candidate boundary locations has decreased to a specified threshold number) is met. In practice, a single merging step has been found to be sufficient. To reduce computational effort while maintaining acceptable results, the merging operation may be omitted. In light of this, the merging operation described with reference to FIGS. 13A and 13B may also be omitted.

[0113] Next, based on the set of candidate boundary locations obtained from each scan trajectory, a boundary is constructed on the fluorescence image, which indicates the boundary between the tumor and healthy tissue.

[0114] In one embodiment, the constructed boundary is a linear curve connecting the peripheral candidate boundary locations. The linear curve in the fluorescence image represents the boundary of the hyperfluorescent region itself. In instances where the fluorescence image is obtained from a tumor containing tissue that fluoresces with a sufficiently specific tracer, the boundary of the hyperfluorescent region is likely to coincide with the tumor edge.

[0115] In another embodiment, the boundary is constructed as a secondary curve that surrounds a primary curve interconnecting the peripheral transition locations and extends outward from the primary curve by a predetermined distance depending on the type of tumor present in the tissue. The secondary curve indicates the area to be treated or ablated. Typically, the secondary curve surrounds the primary curve by a predetermined distance depending on the type of tumor tissue. In one embodiment, the distance between the secondary curve and the primary curve is locally less than a predetermined distance to avoid intersection with certain anatomical structures.

[0116] FIG. 17 shows an example of a boundary B constructed in a fluorescence image FI, in which the boundary construction unit 155 directly constructed the boundary from the set of candidate boundary positions {pt} a (see FIG. 16 ) acquired from the candidate boundary position filter unit 150. The boundary construction unit 155 applied the MatLab function boundary (x, y, s) (s = 0.5) to the filtered set of candidate boundary positions {pt} a (see FIG. 16 ) acquired from the candidate boundary position filter unit 150. This function constructs a boundary that connects the surrounding candidate boundary positions to each other. The shape of the constructed boundary is somewhere between a shape that fits exactly to all the surrounding candidate boundary positions and a convex hull that encloses the surrounding candidate boundary positions.

[0117] Example II

[0118] Figures 18A,…,18F show example fluorescence images obtained in vivo from patient tissue. In this case, the tissue was penile squamous cell carcinoma and was fluoresced by the tracer cetuximab-IRDye800CW.

[0119] 18A shows the filtered set of candidate boundary positions {pt} a output from the candidate boundary position filter unit 150 superimposed on the original fluorescence image FI. FIG. 18B shows the boundary B constructed directly from the filtered set of candidate boundary positions {pt} a obtained from the candidate boundary position filter unit 150 superimposed on the fluorescence image FI.

[0120] FIG. 18C shows a set of candidate boundary positions {pt} c obtained from the further candidate boundary position matching unit 152 superimposed on the fluorescence image.

[0121] FIG. 18D shows a boundary B constructed from the set of candidate boundary locations {pt} c shown in FIG. 18C superimposed on the fluorescence image.

[0122] FIG. 18E shows a set of candidate boundary positions {pt} n acquired from the candidate boundary position integrating unit 153 superimposed on a fluorescent image.

[0123] FIG. 18F shows a boundary B constructed from the set of candidate boundary locations {pt} n shown in FIG. 18E superimposed on the fluorescence image.

[0124] 19A, . . . , 19F show further exemplary results obtained for fluorescence images obtained ex vivo from intact tissue excised from a patient.

[0125] 19A shows the filtered set of candidate boundary positions {pt} a output from the candidate boundary position filter unit 150 superimposed on the original fluorescence image FI. FIG. 19B shows the boundary B constructed directly from the filtered set of candidate boundary positions {pt} a obtained from the candidate boundary position filter unit 150 superimposed on the fluorescence image FI.

[0126] FIG. 19C shows a set of candidate boundary positions {pt} c obtained from the further candidate boundary position matching unit 152 superimposed on the fluorescence image.

[0127] Figure 1 9D shows the boundary B constructed from the set of candidate boundary locations {pt} c shown in Figure 19C superimposed on the fluorescence image.

[0128] FIG. 19E shows a set of candidate boundary positions {pt} n acquired from the candidate boundary position integrating unit 153 superimposed on a fluorescent image.

[0129] FIG. 19F shows a boundary B constructed from the set of candidate boundary locations {pt} n shown in FIG. 19E superimposed on the fluorescence image.

[0130] 20A, . . . , 20F show further exemplary results obtained for fluorescence images obtained ex vivo from excised tissue sections.

[0131] 20A shows the filtered set of candidate boundary positions {pt} a output from the candidate boundary position filter unit 150 superimposed on the original fluorescence image FI. FIG. 20B shows the boundary B constructed directly from the filtered set of candidate boundary positions {pt} a obtained from the candidate boundary position filter unit 150 superimposed on the fluorescence image FI.

[0132] FIG. 20C shows a set of candidate boundary positions {pt} c obtained from the further candidate boundary position matching unit 152 superimposed on the fluorescence image.

[0133] FIG. 20D shows a boundary B constructed from the set of candidate boundary locations {pt} c shown in FIG. 20C superimposed on the fluorescence image.

[0134] FIG. 20E shows a set of candidate boundary positions {pt} n acquired from the candidate boundary position integrating unit 153 superimposed on a fluorescent image.

[0135] FIG. 20F shows a boundary B constructed from the set of candidate boundary locations {pt} n shown in FIG. 20E superimposed on the fluorescence image.

[0136] From the illustrative results shown in this example, we conclude that the boundary B constructed from the set of filtered candidate boundary locations {pt} a best matches the tumor margin indicated by the pathologist. For example, for the in vivo fluorescence image, the constructed boundary shown in Figure 18B best matches the pathologist's analysis.

[0137] Example 3

[0138] 21A, . . . , 21F show exemplary results obtained for fluorescence images obtained in vivo from patient tissue containing head and neck squamous cell carcinoma that fluoresced with the tracer Cetuximab-IRDye800CW.

[0139] 21A shows the filtered set of candidate boundary locations {pt} a output from the candidate boundary location filter unit 150 superimposed on the original fluorescence image F I. FIG. 21B shows the boundary B constructed directly from the filtered set of candidate boundary locations {pt} a obtained from the candidate boundary location filter unit 150 superimposed on the fluorescence image.

[0140] FIG. 21C shows a set of candidate boundary positions {pt} c obtained from the further candidate boundary position matching unit 152 superimposed on the fluorescence image.

[0141] FIG. 21D shows a boundary B constructed from the set of candidate boundary locations {pt} c shown in FIG. 21C superimposed on the fluorescence image.

[0142] FIG. 21E shows a set of candidate boundary positions {pt}n acquired from the candidate boundary position integrating unit 153 superimposed on a fluorescent image.

[0143] FIG. 21F shows a boundary B constructed from the set of candidate boundary locations {pt} n shown in FIG. 21E superimposed on the fluorescence image.

[0144] 22A, . . . , 22F show further exemplary results obtained for fluorescence images obtained ex vivo from intact tissue excised from a patient.

[0145] 22A shows the filtered set of candidate boundary positions {pt} a output from the candidate boundary position filter unit 150 superimposed on the original fluorescence image FI. FIG. 22B shows the boundary B constructed directly from the filtered set of candidate boundary positions {pt} a obtained from the candidate boundary position filter unit 150 superimposed on the fluorescence image FI.

[0146] FIG. 22C shows a set of candidate boundary positions {pt} c obtained from the further candidate boundary position matching unit 152 superimposed on the fluorescence image.

[0147] FIG. 22D shows a boundary B constructed from the set of candidate boundary locations {pt} c shown in FIG. 22C superimposed on the fluorescence image.

[0148] FIG. 22E shows a set of candidate boundary positions {pt} n acquired from the candidate boundary position integrating unit 153 superimposed on a fluorescent image.

[0149] FIG. 22F shows a boundary B constructed from the set of candidate boundary locations {pt} n shown in FIG. 22E superimposed on the fluorescence image.

[0150] 23A, . . . , 23F show further exemplary results obtained for fluorescence images obtained ex vivo from excised tissue sections.

[0151] 23A shows the filtered set of candidate boundary locations {pt} a output from the candidate boundary location filter unit 150 superimposed on the original fluorescence image F I. FIG. 23B shows the boundary B constructed directly from the filtered set of candidate boundary locations {pt} a obtained from the candidate boundary location filter unit 150 superimposed on the fluorescence image.

[0152] FIG. 23C shows a set of candidate boundary positions {pt} c obtained from the further candidate boundary position matching unit 152 superimposed on the fluorescence image.

[0153] FIG. 23D shows a boundary B constructed from the set of candidate boundary locations {pt} c shown in FIG. 23C superimposed on the fluorescence image.

[0154] FIG. 23E shows a set of candidate boundary positions {pt} n acquired from the candidate boundary position integrating unit 153 superimposed on a fluorescent image.

[0155] FIG. 23F shows a boundary B constructed from the set of candidate boundary locations {pt} n shown in FIG. 23E superimposed on the fluorescence image.

[0156] From the exemplary results shown in Example III, we conclude that the boundary B constructed from the filtered set of candidate boundary locations {pt} a best matches the tumor margin as indicated by the pathologist. For example, for a fluorescent image obtained in vivo, the constructed boundary shown in Figure 21B best matches the pathologist's analysis.

[0157] Example IV

[0158] Figures 24A–24F show the fluorescence imaging results obtained from tongue cancer tissue sections fluoresced with the tracer ONM-100. ONM-100 consists of polymeric micelles labeled with indocyanine green (ICG). Chemically, the ONM-100 drug substance consists of a diblock copolymer of polyethylene glycol (PEG) (approximately 113 repeating units) and a poly(methyl methacrylate) derivative covalently bound to functionalized ICG as a fluorophore. The ICG content was determined by a qualified method, with a molecular weight of 37.5 ± 12.5 kD.

[0159] 24A shows the filtered set of candidate boundary positions {pt} a output from the candidate boundary position filter unit 150 superimposed on the original fluorescence image FI. FIG. 24B shows the boundary B constructed directly from the filtered set of candidate boundary positions {pt} a obtained from the candidate boundary position filter unit 150 superimposed on the fluorescence image FI.

[0160] FIG. 24C shows a set of candidate boundary positions {pt} c obtained from the further candidate boundary position matching unit 152 superimposed on the fluorescence image.

[0161] FIG. 24D shows a boundary B constructed from the set of candidate boundary locations {pt} c shown in FIG. 24C superimposed on the fluorescence image.

[0162] FIG. 24E shows a set of candidate boundary positions {pt} n acquired from the candidate boundary position integrating unit 153 superimposed on a fluorescent image.

[0163] FIG. 24F shows a boundary B constructed from the set of candidate boundary locations {pt} n shown in FIG. 24E superimposed on the fluorescence image.

Claims

1. An image processing device (1) for processing a fluorescence image obtained from tissue of a subject irradiated with excitation light that has been made photosensitized by a fluorescent agent, the fluorescence image (FI) comprising an array of pixels having respective fluorescence signal values. The image processing device is configured to perform the following operations to indicate in the fluorescence image a boundary between a region of interest representing a portion of tissue containing a tumor and a reference region outside the region of interest: obtaining a respective fluorescence signal vector (FL(...)) for each scanning trajectory (L), wherein each value of each fluorescence signal vector (FL(...)) is an indication of the magnitude of the fluorescence signal in the fluorescence image (FI) at each position (p) of the scanning trajectory (L); The scanning trajectory (L) extends through the tissue area (TA). Each modified contrast-to-noise ratio vector (CNR LM (...) ) is evaluated for each position (p) of the scanning trajectory (L). Each value of each contrast-to-noise ratio vector (CNR LM (...) ) is calculated for each position (p) of the scanning trajectory (L) as follows: Here, FB is the reference fluorescence signal value which is the average value of the reference fluorescence signal values ​​of the fluorescence image, S is the standard deviation of the reference fluorescence signal values, and c is a predetermined constant. Here, the image processing device is configured to repeat the following steps for each of a plurality of positions along the scanning trajectory. A position (ps) from a number of positions along the scan trajectory is tentatively assigned as a boundary estimate point that indicates the boundary of the region of interest. Calculate the average of the fluorescence signal values ​​of FL (...) to obtain the quantity FB0. The average of the fluorescence signal values ​​of the fluorescence signal vector (FL(...)) corresponding to the positions of the scanning trajectory on the second opposite side to the first side of the provisionally assigned position is calculated to obtain the quantity FB1. If FB0 > FB1, the tumor is presumed to be represented on the first side, the value FB 1 represents the reference fluorescence signal value FB, and the standard deviation (S) is the standard deviation of the fluorescence signal value on the second side. If FB0 < FB1, the tumor was presumed to be present on the second side, where the value FB0 represents the baseline fluorescence signal value FB, and the standard deviation (S) is the standard deviation of the fluorescence signal value on the first side. If the modified contrast-to-noise ratio vector (CNR LM (...) ) has a zero crossing at the provisionally assigned location, the provisionally assigned location is identified as a candidate boundary location. The boundary is indicated based on the set of candidate boundary locations obtained for each scan trajectory.

2. The image processing device (1) according to claim 1 is configured to show the boundary as a linear curve (B) connecting the transition positions around the transition position with each other.

3. The image processing device (1) according to claim 1, configured to show the boundary as a quadratic curve (B') enclosing a primary curve interconnecting the curves around the transition position and extending a certain distance outside the primary curve depending on the type of tumor present in the tissue.

4. 4. An image processing device (1) according to claim 3, configured to construct the quadratic curve (B') in such a way as to avoid intersections with designated anatomical structures.

5. 5. The image processing device (1) according to claim 1, wherein the indicator FL(p) of the magnitude of the fluorescence signal in the fluorescence image (FI) at the position (p) of the scanning trajectory (L) is the average value of the fluorescence signal values ​​of pixels in the fluorescence image (FI) within a one-dimensional window that includes the position (p) and is oriented laterally with respect to the direction of the scanning trajectory.

6. 6. The image processing device (1) according to claim 5, wherein the average value is a weighted sum of the fluorescence signal values ​​of the pixels in a one-dimensional window according to a Gaussian function having a maximum value at the position (p) of the scanning trajectory (L).

7. 2. The image processing device (1) according to claim 1, further configured to perform low-pass filtering of the modified contrast-to-noise ratio vector.

8. In the image processing device (1) according to any one of claims 1 to 7, each of the scanning trajectories includes at least two scanning trajectories having mutually different directions.

9. 4. An image processing device (1) according to claim 1, 2 or 3, wherein the fluorescence image (FI) is obtained from a tissue sample taken from the object and placed on a background, and the image processing device (1) is configured to identify an area in the fluorescence image representing the tissue sample as the tissue area and to identify an area in the fluorescence image (FI) of the background as the background area.

10. 10. The image processing device (1) according to claim 9, further configured to determine a first maximum intensity in a tissue region of the fluorescence image (FI), determine a second maximum intensity along a scanning trajectory, and skip further processing steps for the scanning trajectory if the second maximum intensity is less than a predetermined percentage of the first maximum intensity.

11. 11. The image processing device (1) according to claim 1, wherein the predetermined constant c is 2.

12. An image processing device (1) according to any one of claims 1 to 11, comprising a camera (2) for acquiring a fluorescence image (FI) from tissue of a subject to which a fluorescent agent has been administered, and a display device (3) for displaying the tissue and the constructed boundary.

13. An image processing device (1) according to claim 1, adapted to exclude candidate boundary locations from further processing if the fluorescence intensity indication is below a threshold value.

14. 14. The image processing device (1) according to claim 13, wherein the indication value is a maximum value of the fluorescence intensity within a window centered on the candidate boundary position.

15. A medical treatment device (100) comprising a camera (2) for acquiring a fluorescence image (FI) from tissue of a subject to which a fluorescent agent has been administered, an image processing device (1) according to any one of claims 1 to 14, and a treatment device (5) for performing a medical treatment to ablate or irradiate a tumor according to the constructed boundary.

Images