Image processing apparatus and medical treatment apparatus including same

Abstract

Disclosed herein is an image processing device (1) for processing a fluorescence image obtained from a tissue of a subject irradiated with excitation light, the tissue being photosensitized with a fluorescent agent, the fluorescence image (FI) comprising an array of pixels having respective fluorescence signal values. The image processing device is configured to identify a transition position where a zero crossing occurs in a modified contrast noise ratio vector (CNRLM (...)), and to indicate a boundary in the fluorescence image representing an edge between a tumor-containing tissue and a healthy tissue based on the identified transition position, wherein the contrast-to-noise ratio vector (CNRLM (...)) is defined as # imgabs0 # wherein FL (...) is a fluorescence signal vector of a scan trajectory L in the fluorescence image FI, each value being indicative of the amplitude of the fluorescence signal in the fluorescence image FI at each position p of the scan trajectory L, FB is a reference fluorescence signal, being an average value of the reference fluorescence signal values of the fluorescence image, S is a standard deviation of the reference fluorescence signal values, and n is an integer greater than or equal to 2. C is a predetermined constant.

Description

Background Art

[0001] The present application relates to an image processing device.

[0002] The present application also relates to a medical treatment device comprising an image processing device.

[0003] Treatment for most types of solid cancers involves radical surgical resection of all tumor tissue. However, distinguishing normal tissue from tumor tissue during surgery remains difficult. Therefore, it is not uncommon for tumor-positive margins to be detected during pathological evaluation two to five days after surgery. According to the literature, the rate of tumor-positive margins (TPM) varies from 10% to 35% depending on the tumor type. See, for example, Orosco, RK, et al., "Positive surgical margins in the 10 most common solid cancers," Sci. Rep. 8, 56-86 (2018). https: / / doi.org / 10.1038 / s41598-018-23403-5. If tumor tissue is present at or near the margins of resected tissue, the risk of local recurrence and distant metastasis increases, implying decreased survival. Therefore, TPM requires additional treatment, such as resurgery, radiation therapy, and / or systemic therapy. Unfortunately, this is associated with increased morbidity and increased psychological burden for the patient. Therefore, being able to correctly visualize tumor tissue during surgery is crucial. However, current optical technologies and the visual and tactile information obtained by the physician are insufficient to fully determine tumor boundaries. Therefore, new technologies that can visualize tumors in real time are being investigated with the goal of reducing the number of TPMs, thereby reducing additional treatments and morbidity.

[0004] Fluorescence molecular imaging (FMI) is an imaging technique that has attracted attention because it can achieve real-time tumor visualization both within the patient's body (i.e., in vivo) and immediately after resection (ex vivo). To this end, the tissue to be examined is prepared with a fluorescent agent (FA) by administering it to the patient or impregnating the tissue with it.

[0005] For example, see the article by Suhail M Odeh et al.: "Skin Lesion Diagnosis Using Fluorescence Images", January 1, 2006 (2006-01-01), "IMAGE ANALYSIS AND RECOGNITION", LECTURE NOTES IN COMPUTERSCIENCE (LNCS), Springer, Berlin, Germany, pp. 648-659.

[0006] This paper proposes a computer-aided diagnosis system for skin lesions. Various parameters or features extracted from fluorescence images are evaluated for cancer diagnosis. The choice of parameters significantly impacts the cost and accuracy of the automated classifier. A genetic algorithm (GA) is used to select these parameters using a K-nearest neighbor (KNN) classifier.

[0007] Fluorescent agents (FAs) can be non-targeted fluorescent dyes, such as indocyanine green (ICG), or targeted fluorescent dyes used to image tumor tissue and infections and track drug treatments. Although FMI studies in the near-infrared (NIR) spectral range (700-900 nm) have shown promising results, some complications have been found to persist. For example, light scattering and absorption by biological components such as water and blood can lead to attenuation of the excitation light, resulting in reduced sensitivity and contrast of the fluorescence image. These factors can cause false-positive TPMs to appear on fluorescence images, even if TPMs are not present in the patient. As a result, the obtained fluorescence images (FI) are not always directly suitable for guidance to surgeons or other medical professionals, or for use in medical treatment devices. Summary of the Invention

[0008] According to a first aspect, there is provided an image processing apparatus as claimed in claim 1 for processing a fluorescence image obtained from tissue of a subject to facilitate a specialist or surgical device in performing a treatment.

[0009] According to a second aspect, a medical treatment device comprising such an image processing device is provided as claimed in claim 15 .

[0010] The image processing device of the first aspect as claimed in claim 1 is configured to process a fluorescent image obtained from tissue of a subject. The tissue is irradiated with excitation light after being photosensitized with a fluorescent agent, and the fluorescent image obtained from the irradiated photosensitive tissue includes an array of pixels with respective fluorescence signal values. The fluorescent agent is used to display different types of tissue, such as tumor tissue and tumor-free tissue. In one example, the fluorescent agent is a targeted fluorescent tracer, such as Cetuximab-IRDye800CW or Hexvix, for imaging tumor tissue and / or infection and tracking drug treatment, which can be administered to a patient or can be used to impregnate tissue. In another example, the fluorescent agent is a non-targeted fluorescent dye, such as indocyanine green (ICG), for imaging tissue perfusion. The excitation light used to irradiate the tissue, whether in vivo or in vitro, is typically in the infrared range.

[0011] Currently, there are different types of imaging systems available on the market. The first type operates in the near-infrared (NIR) spectral range (700-900nm). The second type operates in the short-wave infrared (SWIR) spectral range (1000-1700nm). Due to more limited autofluorescence and significantly reduced scattering, the second type of imaging system is believed to produce higher-contrast images and have deeper tissue penetration.

[0012] The image processing apparatus provided herein is configured to perform the following operations to determine an indication of a boundary between a target region representing a portion of tissue containing a tumor and a reference region outside the target region in a fluorescent image:

[0013] The image processing device obtains individual fluorescence signal vectors for each scanning trajectory. Each series of values ​​in a fluorescence signal vector indicates the amplitude of the fluorescence signal in the fluorescence image at each position along the scanning trajectory. For computational efficiency, the scanning trajectory extends along a straight line and can therefore be referred to as a scan line. However, other scanning trajectories, such as curved trajectories, are also contemplated.

[0014] The image processing device then evaluates the signal vectors F for each fluorescence signal L Each corrected contrast-to-noise ratio vector CNR of (...) LM (...). For each scanning trajectory L, the contrast-to-noise ratio vector CNR LM The value of (...) is calculated for each position p of the scanning trajectory L as follows:

[0015]

[0016] Among them F B is the reference fluorescence signal, is the average value of the reference fluorescence signal values ​​of the fluorescence image, S is the standard deviation of the reference fluorescence signal values ​​obtained from the background, and c is a predetermined constant.

[0017] In one embodiment of the image processing apparatus, the predetermined constant c used to determine the corrected contrast-to-noise ratio is approximately 2. The result obtained thereby closely matches the boundaries identified by a pathologist when examining the tissue. Alternatively, a value slightly less than 2 may be selected, for example, in the range of 1.5 to 2. In this case, the indicated boundaries will extend beyond the boundaries identified by the pathologist, thereby providing a safety margin.

[0018] According to another method, the contrast-to-noise ratio CNRL(p) is determined as follows:

[0019]

[0020] It is then determined at which positions along line L the contrast-to-noise ratio CNR L The value of (p) crosses the threshold 2 / c.

[0021] The image processing device identifies a set of transition positions pt along each scan trajectory, where the contrast-to-noise ratio vector CNR is modified. LM (...) has zero crossings.

[0022] The image processing apparatus generates a candidate boundary position set based on the identified transition position set.

[0023] The image processing apparatus generates an indication of a boundary between the target region and the reference region based on a set of candidate boundary positions obtained for each scanning trajectory.

[0024] In one embodiment, the image processing device (IPD) is configured to indicate the boundary as a master curve interconnecting peripheral candidate boundary locations.

[0025] In another embodiment, the image processing device is configured to indicate the boundary as a secondary curve that encloses a primary curve interconnecting peripheral candidate boundary locations and extends a distance outside the primary curve that depends on the type of tumor present in the tissue. The distance to be selected, i.e., the guideline for the tumor-free margin, depends on the tumor type, as described in Table 1 of Voskuil et al., "Intraoperative imaging in pathology-assisted surgery," Nat Biomed Eng, 6 (2022) 503, https: / / doi.org / 10.1038 / s41551-021-00808-8. In a further refinement of this embodiment, the image processing device is configured to construct the secondary curve in a manner that avoids intersection with specified anatomical structures.

[0026] In one embodiment of the image processing apparatus, the indication of the fluorescence signal amplitude in the fluorescence image at a scanning trajectory location is an average of the fluorescence signal values ​​of pixels within a one-dimensional window in the fluorescence image that includes the location and is transverse to the scanning trajectory. This operation produces an amplitude indication with an improved signal-to-noise ratio, particularly when the scanning trajectory substantially crosses the boundary to be determined transversely. In one specific example, the average is determined by taking a weighted sum of the fluorescence signal values ​​of the pixels within the one-dimensional window according to a Gaussian function that has a maximum value at the scanning trajectory location.

[0027] In one embodiment, the image processing apparatus is configured to repeat the following steps for each of a plurality of positions along the scanning trajectory.

[0028] The image processing device temporarily designates one position from among a plurality of positions along the scanning trajectory as a hypothetical point of the boundary of the target area.

[0029] The image processing device calculates a reference fluorescent signal value as an average value of fluorescent signal values ​​of fluorescent signal vectors corresponding to positions on one side of the temporarily specified position of the scanning trajectory.

[0030] The image processing device calculates the standard deviation as the standard deviation of the fluorescence signal values ​​along one side of the scanning trajectory.

[0031] In one example, the average fluorescence signal values ​​for the scanning track positions on each side of the temporarily designated position are calculated, and the side with the lowest average fluorescence signal value is assumed to be in the reference region. Therefore, the lowest average fluorescence signal value is used as the reference fluorescence signal value, and the standard deviation used is the standard deviation of the fluorescence signal values ​​for the side of the scanning track with the lowest average fluorescence signal value.

[0032] If the modified contrast-to-noise ratio vector has a zero-crossing point at the temporarily specified position, the image processing apparatus identifies the temporarily specified position as a candidate boundary position.

[0033] In this embodiment, the plurality of positions may generally include all positions of the scan trajectory, except for its ends, so that at least two signal values ​​can be used to estimate the standard deviation. In one example, the image processing device is configured to perform low-pass filtering of the modified contrast-to-noise ratio vector. This mitigates the occurrence of false zero crossings.

[0034] In one embodiment of the image processing device, each scanning trajectory includes at least two scanning trajectories having mutually different directions. A candidate boundary position will generally indicate a position where the scanning trajectory crosses the boundary in a direction transverse to the scanning trajectory. Therefore, a more complete set of candidate boundary positions is obtained by scanning trajectories having mutually different directions. For example, the scanning trajectory includes a first set of scanning trajectories in a horizontal direction and a second set of scanning trajectories in a vertical direction. This embodiment is very efficient in terms of computation. At the expense of increasing the amount of computation, additional candidate boundary positions can be calculated using more scanning trajectory directions. For example, a set of scanning trajectories can be used for each direction that is a multiple of n degrees, where n is, for example, 10.

[0035] As described above, the image processing device can be used for both in vivo and in vitro analysis. In the case where the image processing device is used for in vitro analysis, the fluorescence image is obtained from a tissue sample obtained from a subject, which is then placed on a background, typically a dark background. In one embodiment, the image processing device is configured to identify an area in the fluorescence image representing the tissue sample as a tissue area, and to identify a background area in the fluorescence image as a background area. In one example of this embodiment, the image processing device is further configured to determine a first maximum intensity in a target area of ​​the fluorescence image, determine a second maximum intensity along a scan trajectory, and skip further processing steps for the scan trajectory if the second maximum intensity is less than a predetermined fraction of the first maximum intensity.

[0036] In one embodiment, the image processing apparatus may also be optionally or additionally configured to reject a candidate boundary location from further processing if the fluorescence intensity indicator value is less than a threshold. The indicator value may be the intensity value of the fluorescence image at the candidate boundary location to be determined. However, during this candidate boundary location filtering operation, noise sensitivity can be reduced by setting the indicator value to the maximum fluorescence intensity within a window centered on the candidate boundary location. This value is indicated by the "movmax" function, where the window size can be selected within the range of 2 to 10 pixels, for example, 5 pixels.

[0037] In one embodiment, an image processing device includes a camera for obtaining a fluorescent image of tissue from a subject and further includes a display device for displaying one or more indications of the tissue and its boundaries, for example, as a primary curve and / or as secondary curves surrounding the primary curve. This embodiment is particularly suitable for guiding a medical professional in removing or irradiating a tumor during treatment. In this case, the patient undergoing treatment has already been administered a fluorescent agent.

[0038] According to a second aspect of the present application, a therapeutic device for treating a subject to which a fluorescent agent has been administered prior to treatment is provided. The therapeutic device includes an excitation light source for irradiating tumor-containing tissue in the subject with excitation light, a camera for obtaining a fluorescent image from the irradiated tissue, an image processing device as described above, and a therapeutic device for removing or reducing the tumor from the subject according to the boundaries indicated by the image processing device. In one example, the therapeutic device is a surgical robot that removes a tumor from tissue. In another example, the therapeutic device is a therapeutic irradiation device that irradiates the tumor. In yet another example, the therapeutic device is a therapeutic irradiation device that activates a therapeutic substance within a specified range of boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] These and other aspects of the present disclosure are described in more detail with reference to the accompanying drawings, in which:

[0040] Figure 1 Schematically illustrates an embodiment of an image processing apparatus according to the first aspect;

[0041] Figure 2 shows an example module in another embodiment thereof;

[0042] Figure 3 Further example modules are shown;

[0043] Figure 4 Shown in more detail Figure 3 Part of the module;

[0044] Figure 5 shows a further embodiment of the image processing apparatus according to the first aspect of the present invention;

[0045] Figure 6 An embodiment of a medical treatment device according to the second aspect is shown;

[0046] Figure 7A shows a fluorescent image captured from tissue containing a tumor;

[0047] Figure 7B The result of the first conversion step applied to the fluorescence image is shown;

[0048] Figure 7C Shows the application Figure 7B The result of the second transformation step of the result shown;

[0049] Figure 8 A first data processing operation is shown;

[0050] Figure 9A and Figure 9B shows the result obtained in the second data processing operation;

[0051] Figure 10A 、 Figure 10B 、 Figure 10C Various aspects of a third data processing operation are shown;

[0052] Figure 11A and Figure 11B shows the results of a fourth data processing operation variant;

[0053] Figure 12A and Figure 12B shows the results of a fifth data processing operation variant;

[0054] Figure 13A and Figure 13B shows the results of a fifth data processing operation variant;

[0055] Figure 14 shows the result of a sixth data processing operation;

[0056] Figure 15 shows the result of a seventh data processing operation;

[0057] Figure 16 shows the result of an eighth data processing operation;

[0058] Figure 17 Shown superimposed on Figure 7A Boundary indication on the fluorescence image;

[0059] 18A to 18F A first example result obtained from a fluorescence image obtained in vivo from patient tissue is shown;

[0060] Figures 19A to 19F shows a second example result obtained from a fluorescence image of the tissue obtained ex vivo following complete resection from the patient;

[0061] 20A to 20F showing a third example result obtained from a fluorescent image obtained in vitro from a slice of the resected tissue;

[0062] Figures 21A to 21F A fourth example result obtained from a fluorescence image obtained in vivo from another patient's tissue is shown;

[0063] Figures 22A to 22F shows a fifth example result obtained from a fluorescence image obtained ex vivo from intact tissue resected from a patient;

[0064] Figures 23A to 23F shows a sixth example result obtained from a fluorescence image obtained ex vivo from a slice of excised tissue;

[0065] 24A to 24F A seventh result obtained from a fluorescent image obtained from a tissue section is shown. DETAILED DESCRIPTION

[0066] Figure 1 An image processing apparatus 1 is schematically shown, configured to process a fluorescence image FI of tissue of a subject. The tissue is photosensitized with a fluorescent agent (e.g., a contrast agent) and irradiated with excitation radiation. The fluorescence image FI obtained thereby comprises an array of pixels having respective fluorescence signal values. The image processing apparatus 1 is configured to determine a boundary between a target region TR representing a portion of tissue containing a tumor and a reference region RR outside the target region in the fluorescence image.

[0067] The image processing apparatus 1 has a signal vector retrieval module 10 configured to obtain each fluorescence signal vector F of each scanning track L indicated by the scanning track selection module 11. L (...). Fluorescence signal vector F L Each value of (...) indicates the amplitude of the fluorescence signal in the fluorescence image FI at each position p of the scanning trajectory L.

[0068] The contrast-to-noise ratio calculation module 12 calculates each fluorescence signal vector F L (...) Calculate each corrected contrast-to-noise ratio vector CNR LM (...). Contrast-to-noise ratio vector CNR LM The corresponding values ​​contained in (...) are calculated for each position p of the scanning trajectory L from the fluorescence signal vector F L (...) is calculated as follows:

[0069]

[0070] The statistical module 13 provides a reference fluorescence signal F B , the signal is the average value of the reference fluorescence signal value of the fluorescence image, and the standard deviation S of the reference fluorescence signal value. In addition, c is a predetermined constant, for example, c=2.

[0071] The zero-crossing detection module 14 identifies the set of transition positions {pt} along each scan trajectory. These are the modified contrast-to-noise ratio vectors CNR LM (...) has a zero crossing position.

[0072] The scanning trajectory selection module 11 selects a plurality of mutually different scanning trajectories, and determines a respective transition position set for each selected scanning trajectory.

[0073] The boundary indication module 15 then indicates the boundary between the target region TR and the reference region RR based on the transition position sets obtained for each scanning trajectory.

[0074] In the illustrated embodiment, the image processing apparatus 1 further comprises a display module 16 which generates a modified fluorescence image FI″ in which the boundary B indicated by the boundary indication module 15 is superimposed on the original fluorescence image FI. If the original fluorescence image FI was acquired in vivo, this image can assist the surgeon during the procedure of resecting a tumor or irradiating the tumor with therapeutic radiation.

[0075] The scanning trajectory selection module 11 then selects a scanning trajectory L from the scanning trajectory set and instructs the signal vector retrieval module 10 to obtain a fluorescence signal vector, each of which indicates a fluorescence signal vector having coordinates xp, y in the fluorescence image FI. p The amplitude of the fluorescence signal at each position p of the scanning trajectory L. In one example, the fluorescence signal vector value corresponds to the amplitude of the fluorescence signal in the fluorescence image FI at each position p of the scanning trajectory L. In another example, the indication FL(p) of the fluorescence signal amplitude in the fluorescence image FI at the position of the scanning trajectory L is the fluorescence image FI including the coordinates xp, y p The average value of the fluorescence signal value of the pixels in a one-dimensional window transverse to the scanning trajectory.

[0076] therefore:

[0077] in

[0078] Where xq = 0, y q = 0 coincides with the position xp, yp. As an example, the weight factors are equal to each other, that is,

[0079] In a preferred embodiment, the average value is a weighted sum of the fluorescence signal values ​​of the pixels within the one-dimensional window according to a Gaussian function or a similar weighting function having a maximum value at position p of the scanning trajectory L.

[0080] When image processing apparatus 1 is used in intraoperative mode, fluorescence image FI is obtained from a tissue sample obtained from a subject and arranged on a background. In one embodiment, image processing apparatus 1 includes a foreground recognition module 17 (indicated by a dashed line), by which image processing apparatus 1 is configured to identify regions in the fluorescence image representing the tissue sample as tissue regions, and to identify background regions in fluorescence image FI as non-tissue regions. In one example of this embodiment, foreground recognition module 17 (indicated by a dashed line) is further configured to determine a first maximum intensity in the tissue region of fluorescence image FI, determine a second maximum intensity along a scan trajectory, and skip further processing steps for that scan trajectory if the second maximum intensity is less than a predetermined fraction of the first maximum intensity.

[0081] exist Figure 1In the embodiment of FIG. 5 , it is assumed that the reference fluorescent signal FB and the standard deviation S are predetermined values.

[0082] Figure 2 An alternative embodiment of the statistics module 13 is shown, which dynamically determines these parameters F B and S as the fluorescence signal vector data F L function.

[0083] The statistics module 13 includes a vector segmentation section 130, which divides the fluorescence signal vector F L (...) is divided into the first part F L0 and Part II F L1 Part IF L0 The fluorescence signal vector F corresponding to the first side of the scanning track relative to the scanning track position ps temporarily specified by the position specifying section 131 is included. L (...) Data. Part II.F L1 The fluorescence signal vector F corresponding to the opposite side of the scanning trajectory relative to the position ps is included L (...) data. Each part may or may not include the fluorescence signal vector F at position ps L The first average value calculation section 132 calculates the fluorescence signal vector F L The first part of (...) L0 Similarly, the second average value calculation section 133 calculates the fluorescence signal vector F L Part 2 of (...) L1 (...) is an average value FB1 of the values. Alternatively, a single section may subsequently calculate these values ​​FB0, FB1. The decision section 134 determines whether the value FB0 exceeds the value FB1. If so, it determines that FB=FB1. Otherwise, it determines that FB=FB0. The decision section 134 also issues a control signal S0 / 1 to the data selector 135. In the first case, FB=FB1, which controls the data selector 135 to select the fluorescence signal vector F L Part 2 of (...) L1 (...) is used as an input to the standard deviation calculation section 136 so as to obtain the fluorescence signal vector F L Part 2 of (...) L1 (...) calculates the standard deviation S of the fluorescence signal data. In the second case, FB=FB0, the control data selector 135 selects the fluorescence signal vector F L The first part of (...) L0(...) as input. In one embodiment, the position specifying section 131 specifies the subsequent position ps of the scanning trajectory, and for each temporarily specified position, the corresponding parameters FB and S are determined and used to calculate the respective modified contrast-to-noise ratios. In addition, if the modified contrast-to-noise ratio vector (CNR LM (...)) have zero crossings at the temporarily specified positions, the zero crossing detection module 14 particularly selects those temporarily specified positions as candidate boundary positions.

[0084] One embodiment of the boundary indication module 15 is Figure 3 Shown in more detail in .

[0085] The boundary indication module 15 shown therein includes a candidate boundary position filtering section 150, which is configured to reject a candidate boundary position for further processing when the indicator value of the fluorescence intensity is less than a threshold value. In one example, the indicator value is the maximum value of the fluorescence intensity within a window centered on the candidate boundary position to be determined. In one example, the maximum value is obtained from an auxiliary image FIa, which is derived from the original fluorescence image once and stored for subsequent reference. The auxiliary image FIa is derived, for example, by a two-pass procedure, wherein in the first pass, for each pixel (x, y) a value FIh(x, y) is determined as:

[0086] FIh(x,y)=max{FI(xw,y),…,FI(x+w,y)}

[0087] And in the second pass:

[0088] FIa(x,y)=max{FIh(x,yw),…,FIh(x,y+w)}

[0089] In one example, the threshold is chosen as a percentage of the maximum value of the complete fluorescence image FI. This percentage can be chosen in the range of 10-80, more preferably in the range of 40% to 60%, depending on the tissue being imaged. For samples with smooth surfaces, such as slices of tissue, 40% is a suitable choice. For samples with higher surface roughness, this percentage should be higher. The window size 2w+1 is chosen in the range of 2 to 10, with 5 being found to be optimal, corresponding to w = 2.

[0090] Figure 3The embodiment of the boundary indication module 15 shown also includes a candidate boundary position matching portion 151, which matches the candidate boundary positions obtained from each pair of scanning trajectories that overlap but have opposite scanning directions. The candidate boundary position matching portion 151 verifies whether the candidate boundary position obtained using the first scanning trajectory of the pair has a corresponding candidate boundary position in the second scanning trajectory of the pair. If the relative positions of the candidate boundary positions in the fluorescent image do not exceed a threshold distance (e.g., 1 or 2 pixels), they are considered to correspond. In addition, the candidate boundary positions may be required to have opposite polarities. That is, if one of them represents a boundary crossing in the direction from the reference area to the target area, the other should represent a crossing in the direction from the target area to the reference area.

[0091] In the illustrated embodiment, the boundary indication module 15 includes a further candidate boundary position matching portion 152 that matches mutually non-overlapping but adjacent candidate boundary position pairs obtained from mutually different scanning trajectories (i.e., having a relative angle other than 0 degrees or 180 degrees). The further candidate boundary position matching portion 152 identifies pairs of first candidate boundary positions obtained from a first scanning trajectory and second candidate boundary positions obtained from a second scanning trajectory that have a Euclidean distance less than a predetermined maximum value (e.g., 3 pixels), and replaces the mutually adjacent candidate boundary position pairs with a single replacement candidate boundary position located at the center between the mutually adjacent candidate boundary positions. It should be noted that in addition to the Euclidean distance metric, different distance metrics may also be used, such as the L1 metric that calculates the sum of distances along the main direction of the image, or the L∞ metric that defines the distance as the maximum value among the main direction distances.

[0092] In the illustrated embodiment, the boundary indication module further includes a candidate boundary position merging portion 153. The candidate boundary position merging portion 153 receives the remaining candidate boundary position set {pt}b from the further candidate boundary position matching portion 152 and merges mutually adjacent positions among the remaining candidate boundary positions. As indicated by the reverse arrow "N", the candidate boundary position merging portion 153 may perform this operation multiple times. The decision portion 154 determines whether the iteration should stop "Y" or continue "N". In one example, the iteration is performed a predetermined number of times, for example 3 to 5 times. In another example, the decision portion 154 determines whether a data-related stopping criterion is met, such as the number of remaining candidate boundary positions being reduced to below a threshold. The threshold may be predetermined or may depend on other information. For example, the threshold is a constant multiplied by the square root of the total number of pixels within the convex hull containing all remaining candidate boundary positions.

[0093] The boundary construction section 155 receives the selected candidate boundary position set {pt}n based on the transition position set from the candidate boundary position merging section 153 and constructs an indication of the boundary B.

[0094] In one example, the boundary construction portion 155 executes the MatLab function boundary(x, y, s). The value s can be selected in the range of 0.1-1, typically in the range of 0.5-1. The best results are obtained with a value selected in the range of 0.8 to 1. However, the default value s=0.5 also provides good results.

[0095] like Figure 3 As shown by the dashed lines in FIG, in some examples, boundary indication module 15 is configured to skip one or more of the operations specified above. For example, boundary construction module 155 directly constructs a boundary from the filtered candidate boundary position set {pt}a obtained by candidate boundary position filtering module 150. In another example, boundary construction module 155 constructs a boundary from the candidate boundary position set {pt}c obtained by candidate boundary position matching module 152.

[0096] In one example, the boundary construction portion 155 is configured to indicate the boundary B as a main curve interconnecting peripheral transition locations.

[0097] In such Figure 4 In another example shown, boundary construction section 155 is configured to construct a boundary as a secondary curve B' that encloses the primary curve at interconnected peripheral transition locations and extends beyond the primary curve by a distance that depends on the type of tumor present in the tissue. To this end, boundary construction section 155 receives a control signal C1 from distance control section 156 specifying a predetermined distance. In response, boundary construction section 155 indicates the boundary as a secondary curve B' that extends beyond the primary curve such that the distance between each point on the secondary curve and the nearest corresponding point on the primary curve is equal to the predetermined distance.

[0098] exist Figure 4 In a variation of the embodiment of the invention, the boundary construction section 155 is further configured to construct the secondary curve in a manner that avoids intersection with a specified anatomical structure. Figure 4 The critical tissue protection portion 157 is shown as a dotted line. The critical tissue protection portion 157 contains information about the location of critical tissues (such as vital organs), and based on this information, the boundary construction portion 155 constructs the secondary curve B' so that it extends outside the main curve B at a predetermined distance by default, but locally extends at a smaller distance to avoid intersecting with the vital organs.

[0099] Figure 5 An embodiment of an image processing device 1 is shown which is particularly suitable for use as a tool during treatment. It is therefore assumed that the subject to be treated has already been administered a fluorescent agent. In this embodiment, the image processing device 1 (e.g. Figure 1The method of claim 1 further comprises a camera 2 for obtaining a fluorescence image FI from tissue of a patient to be treated, and a display device 3 for displaying the tissue and a boundary indication B as shown in a modified fluorescence image FI". In one variant, the image processing device 1 displays the above-mentioned secondary curve B' superimposed on the fluorescence image, optionally in combination with an indication of the main curve B. In the example shown, the image processing device 1 is further configured to emit a control signal C4 to control an excitation light source 4, which is, for example, a near-infrared (NIR) or short-wave infrared (SWIR) light source. The image processing device 1 can thus control the wavelength and intensity of the excitation light source 4 to optimize the contrast in the fluorescence image FI. Alternatively, a separate excitation light source 4 can be used with predetermined settings that are known to give good results.

[0100] Figure 6 An embodiment of a medical treatment device 100 is shown, which is suitable for automatically performing medical treatment on a patient to treat a tumor present in tissue. Figure 5 In the case of the subject to be treated, it is assumed that the subject has been administered with a fluorescent agent. The medical treatment device 100 includes an image processing device 1 (e.g. Figure 1 In addition to the embodiment or its variant), it also includes a camera 2 for obtaining a fluorescence image FI from tissue with a tumor, and a medical treatment device 5 for resecting or irradiating the tumor according to the indicated boundary or activating a therapeutic substance within a specified range of the indicated boundary.

[0101] like Figure 5 In an embodiment, the image processing device 1 may further be configured to emit a control signal C4 to control an excitation light source 4, such as a near-infrared (NIR) or short-wave infrared (SWIR) light source. Even if the medical treatment device 100 is suitable for automatically performing medical treatment, its operation may need to be monitored. For this purpose, a display device 3 as shown by the dotted line may be provided, which displays the modified fluorescent image FI". In addition, the medical treatment device 5 may provide an information signal I5 to the display device 3 to indicate the progress of the treatment. At the same time, the medical treatment device 5 may be connected to a user interface 6, through which the medical expert can fully or partially control the operation of the medical treatment device 5 by means of the control signal C5.

[0102] Example I

[0103] As an example, Figure 7A A fluorescence image FI captured from tissue containing a tumor arranged on a black background is shown. In one example, the fluorescence image has a width Nx of 400 pixels, a height Ny of 600 or 700 pixels, and a resolution of the order of 100 microns per pixel, such as 85 microns per pixel.

[0104] In the example shown, the tissue from which the fluorescence image was obtained was a slice of a tissue volume removed from a subject. In this case, the tumor was penile cancer, and the subject had been administered Cetuximab-IRDye800CW as a fluorescent agent prior to treatment.

[0105] Alternatively, fluorescence images are obtained from the resected tissue volume, i.e., the resection specimen itself.

[0106] In subsequent operations, the image processing apparatus 1 identifies the first region representing the tissue sample in the fluorescence image as the tissue region, and identifies the background region in the fluorescence image FI as the background region BA. In one example, these operations are performed by the foreground recognition module 17 .

[0107] In a first exemplary operation, the foreground recognition module 17 converts the fluorescence image FI into a binary image, such as Figure 7B As shown, in the fluorescence image, the pixels whose intensity exceeds the threshold have a first binary value, Figure 7B , are represented as white pixels in the image, while the remaining pixels have a second binary value and are represented as dark pixels in the image. The threshold selected for this purpose depends on the choice of input device providing the fluorescence image FI. For the PEARL imaging system, the threshold can be selected in the range of 0.001-0.01, for example, a threshold of 0.005. For the SurgVision imaging system, the threshold can be selected in the range of 500-2000, for example, a threshold of approximately 800.

[0108] The region formed by pixels having the first binary value is called a candidate foreground region, and the region formed by pixels having the second binary value is called a candidate background region.

[0109] Subsequently, the foreground identification module 17 performs a dilation operation to expand the candidate foreground region. Usually, a dilation of one pixel is sufficient to avoid occasional loss of image data at the edge of the tissue. Figure 7B As shown, due to noise, in addition to a single large candidate foreground region, there are also multiple small candidate foreground regions. To alleviate this phenomenon, the foreground identification module 17 performs an additional operation in which it selects the largest one among the candidate foreground regions as the foreground region TA representing the tissue. Figure 7C The results shown. Figure 7C The resulting binary image is used as a mask to indicate the boundary between the foreground region representing the tissue and the remaining background region in the fluorescence image FI.

[0110] In one example, the foreground identification module 17 also determines the foreground area (in Figure 7CThe maximum intensity Imax,fg of the fluorescence image FI is represented by TA in FIG. It is also possible to consider determining the maximum intensity of a complete area of ​​the fluorescence image FI. However, this will be more sensitive to noise because even in the background area, isolated pixels may have high intensities.

[0111] like Figure 8 As shown, the scan line selection module 11 constructs at least one scan line that passes through the foreground area TA (such as Figure 7C The scanning trajectory L is defined by the boundary with the background area (as shown). In the example shown, the scanning trajectory is a line extending horizontally from the first boundary with the background at position x0 to the second boundary with the background at position x1. Based on the indication of the scanning trajectory L, the signal vector retrieval module 10 obtains the fluorescence signal vector F L (...). Fluorescence signal vector F L Each value of (...) indicates the amplitude of the fluorescence signal in the fluorescence image FI at each position p of the scanning trajectory L. The constructed scanning trajectory can represent a strip with a width exceeding 1 pixel. In this case, the fluorescence signal vector F L Each value of (...) is a weighted sum of the intensities of the pixels within the strip width. That is, if the line extends in a first direction x and the strip has width in direction y, the weighted sum of the pixels within the strip width with the same x coordinate is calculated. The weighted sum can be a simple average with the same weighting factor for each pixel, but it can also be a Gaussian weighting function. If the foreground area is concave, more tracks can be constructed on the same line, each track having its own pair of endpoints at respective pairs of points on the boundary. Experiments were performed with different strip width values ​​ranging from 3 to 21 pixels. The best results were obtained with a strip width of 11 pixels, but other choices within this range were found to be suitable.

[0112] In one example, the scanning line selection module 11 determines the fluorescence signal vector F of each scanning track L L (...) values, denoted as Imax,ln. In this example, it is also determined whether the maximum intensity Imax,ln is at least a predetermined percentage, e.g., approximately 10%, of the maximum intensity Imax,fi of the foreground region of the entire fluorescence image. Experiments were conducted with various values ​​of the predetermined percentage, selected from 0, 1, 10, 33, and 50%. Optimal results were achieved with a predetermined percentage of 33.

[0113] If, in this example, the fluorescence signal vector F LIf the maximum intensity Imax,ln found in the values ​​of (...) is less than a predetermined percentage of the maximum intensity Imax,fi, the scan track L is not selected for further processing. This deselection is based on the assumption that the lack of high-intensity pixels along the scan track indicates the absence of tumor tissue in the tissue path corresponding to the scan track. The predetermined percentage can be selected in the range of 10% to 50%. Experiments have shown that 40% is optimal.

[0114] However, if the fluorescence signal vector F L The maximum intensity Imax,ln found in the values ​​of (...) is at least equal to a predetermined percentage of the maximum intensity Imax,fi, and the contrast-to-noise ratio calculation module 12 is the fluorescence signal vector F L (...) Calculate the corrected contrast-to-noise ratio vector CNR LM (...). Based on the modified contrast-to-noise ratio vector CNR LM (...) data, estimating one or more locations where the scanning trajectory crosses the boundary between the healthy tissue subregion and the tumor tissue subregion. The location where the scanning trajectory crosses the boundary in the direction from the healthy tissue subregion to the tumor tissue subregion is characterized by a modified contrast-to-noise ratio vector CNR LM (...) a zero crossing from the negative polarity to the positive polarity, and the position where the scanning trajectory crosses the boundary from the tumor tissue subregion to the healthy tissue subregion is characterized by a zero crossing from the positive polarity to the negative polarity.

[0115] In one example, the CNR calculation module 12 calculates the CNR vector CNR based on the background intensity FB and the respective predetermined values ​​of the standard deviation S. LM (...). This implementation has the advantage of relatively low computational complexity. Another implementation that can more accurately determine the boundary consists of the following procedure, which is repeated for each pixel xs along the scan trajectory, excluding the endpoints of the scan trajectory. For simplicity, it is assumed that the scan trajectory extends in the x direction, and the boundaries between the foreground and background are x0 and x1, respectively. However, this description is also applicable to any direction.

[0116] In this improved procedure, it is assumed that the selected pixel xs coincides with the boundary, and all pixels on a first side of the selected pixel xs on the scanning trajectory represent tumor tissue, while all pixels on a second side of the selected pixel on the scanning trajectory (opposite to the first side) represent healthy tissue.

[0117] Based on this assumption, the vector segmentation section 130 divides the fluorescence signal vector F L (...) is divided into the first part F L0 and Part II F L1 Part IF L0represents the portion of the scan trajectory between the first endpoint (here, the x-coordinate x0) and the temporarily specified boundary position xs. The second portion F L1 It represents the portion of the scanning trajectory between the temporarily specified boundary position xs and the second endpoint (here, the x-coordinate x1).

[0118] Based on this assumption, the first average value calculation section 132 calculates the fluorescence signal vector F L The first part of (...) L0 Similarly, the second average value calculation section 133 calculates the fluorescence signal vector F L Part 2 of (...) L1 The average value of the value of FB1.

[0119] Thus, if ps (here xs) is a temporarily selected position on the scan trajectory, and p0 (here x0) and p1 (here x1) are the positions of the end pixels, then the average value of the intensity on each side of the position ps is determined as follows:

[0120]

[0121] The tumor is considered to be represented in the pixels of the scanning trajectory on the side with the highest average intensity ps. The pixels on the scanning trajectory on the opposite side with the lowest average intensity are considered to represent the background. This is determined by the decision section 134.

[0122] Based on this assumption, the mean value FB and the standard deviation S of the pixel values ​​of the pixels on the scanning trajectory on the second side of the selected pixel are calculated.

[0123] Therefore, if FB0>FB1, it is assumed that the tumor is represented in the range p0-ps, and the value FB1 represents the assumed average background intensity level FB. In this case, the decision section 134 causes the data selector 135 to select the fluorescence signal vector F by selecting the signal S0 / 1. L Part 2 of (...) L1 as input to the standard deviation calculation section 136. Therefore, in this case, the standard deviation S is estimated to be:

[0124]

[0125] In addition, if FB0≤FB1, it is assumed that the tumor is represented in the range ps-p1, and the value FB0 represents the assumed average background intensity level FB. In this case, the determination section 134 causes the data selector 135 to select the fluorescence signal vector F by selecting the signal S0 / 1. L The first part of (...) L0 as input to the standard deviation calculation section 136. Therefore, in this case, the standard deviation S is estimated to be:

[0126] Based on the values ​​of the average background intensity FB and the standard deviation S estimated for the temporarily specified position ps, the contrast-to-noise ratio calculation module 12 calculates the corrected contrast-to-noise ratio CNR LM (p) is:

[0127]

[0128] Thus, for each assumed ps we obtain a vector CNR LM (...), we get a matrix where each row represents the CNR calculated for a specific temporally specified position ps LM (...) value.

[0129] As an example, Figure 9A The values ​​of the fluorescence signal vector FL(p) along the scanning trajectory are shown, along with the average values ​​FB0 and FB1 of the first and second parts of the fluorescence signal vector FL(p), calculated based on a temporarily selected position ps. In this example, the value FB1 is smaller than the value FB0, and it is assumed that the portion p0-ps extends within the image region representing the tumor, while the portion ps-p1 extends within the region representing healthy tissue. Therefore, the value FB1 is selected as the background value FB, and the standard deviation S is calculated from the values ​​of the fluorescence signal vector FL(p) within the ps-p1 range.

[0130] Thus Figure 9B As shown, the corrected contrast-to-noise ratio vector CNR is calculated from the value of the fluorescence signal vector FL(p) LM (...).

[0131] Based on this input, the zero-crossing detection module 14 determines for which values ​​of the temporarily specified position ps the corrected contrast-to-noise ratio vector CNR is obtained for this position ps LM (...) does have a zero crossing point. The corresponding position of the scanning trajectory in the fluorescence image FI is considered to represent the candidate boundary position. Figure 9A 、 Figure 9B In the example shown, we can see the modified contrast-to-noise ratio vector CNR obtained for the temporarily specified position ps LM (...) There is no zero crossing at this position ps. Therefore, this position ps is not considered as a candidate boundary position.

[0132] It should be noted that each modified contrast-to-noise ratio vector CNR can be processed before identifying the zero-crossing points therein. LM (...) Apply a low-pass filter to eliminate false zero crossings caused by noise. In one example, the low-pass filter has a normalized passband frequency less than 0.5, typically less than 0.1. Best results are achieved using a normalized passband frequency of 0.045.

[0133] Therefore, only the modified contrast-to-noise ratio vector CNR that coincides with the temporary specified position ps is determined. LM It is sufficient to determine whether there is a zero crossing at position p of (...). That is, the modified contrast-to-noise ratio vector CNR is calculated only in the neighborhood of position ps. LM (...) is sufficient, optionally taking into account the window of the low-pass filter. However, the complete calculation of each modified contrast-to-noise ratio vector CNR LM (...) may be advantageous so that subsequent operations can be easily performed in parallel using a vector processor.

[0134] It should be noted that the output of the zero-crossing detection module 14 may include a set of multiple candidate boundary locations {pt}. The set of candidate boundary locations may include a first candidate boundary location where the scanning trajectory crosses the boundary from the image region representing healthy tissue to the image region representing tumor tissue, and a second candidate boundary location where the scanning trajectory crosses the boundary from the image region representing tumor tissue to the image region representing healthy tissue. If the scanning trajectory does not enter the region representing non-healthy tissue, the set may be empty. In some cases, the boundary between the image region representing tumor tissue and the image region representing healthy tissue is concave. In these cases, the zero-crossing detection module 14 may identify multiple pairs of candidate boundary locations for a single scanning trajectory.

[0135] like Figure 10A As shown, in operation, the image processing device 1 performs the above-described procedure on a plurality of scanning tracks L1, ...Ln. The distance between the scanning tracks is generally related to the strip width described above. If the strip width is larger, the scanning tracks can be further apart. For example, the distance between the scanning tracks can be selected to be equal to the strip width.

[0136] In some embodiments, all scan trajectories are parallel. However, this complicates the detection of tissue boundaries that are in the same direction as the scan trajectories.

[0137] This can be avoided by performing the procedure using at least two sets of scanning trajectories having mutually different directions. In one example, the subsequent sets of scanning trajectories have directions that differ by 45 degrees. Figure 10A 、 Figure 10B 、 Figure 10CThree such sets are shown. In another example, the procedure is performed using two sets of scanning trajectories with mutually orthogonal directions. In this case, the directions x, y are chosen to correspond to the directions of the main axes in the fluorescence image, which is preferred for computational simplicity. In addition, the procedure can be performed for more sets of scanning trajectories, for example at angles of 0, 10, 20... degrees relative to the main axes of the fluorescence image. The procedure can be performed using a larger set of scanning angles, for example scanning angles that differ by 1 degree from each other. However, this is not preferred because it involves an additional computational burden and does not substantially improve the performance in terms of accuracy. For practical purposes, a difference of 10 degrees was found to be optimal.

[0138] The total number of pixels of the scan trajectory should cover at least a predetermined fraction of the total number of pixels in the foreground area.

[0139] If the scan track represents a strip of pixels, the predetermined fraction may be relatively small.

[0140] The zero-crossing detection module 14 repeats the zero-crossing estimation procedure for all scanning trajectories to obtain a plurality of candidate boundary positions of the boundary between healthy tissues and non-healthy tissues. Figure 11A The candidate boundary positions obtained by using 36 sets of scanning trajectories with angles differing by 10 degrees are shown superimposed on the original fluorescence image FI. Figure 11B Alternative results are shown using four sets of scanning trajectories with angles differing by 90 degrees.

[0141] Based on the set of candidate boundary positions obtained from the zero-crossing detection module 14 , the boundary indication module 15 generates an indication of the boundary between the region representing the tumor tissue and the healthy tissue region in the image.

[0142] In one example, the boundary indication module 15 compares the candidate boundary positions of each scanning trajectory with the candidate boundary positions of each overlapping opposite direction scanning trajectory. In this example, the boundary indication module 15 determines which candidate boundary positions of the first scanning trajectory match the candidate boundary positions of the overlapping opposite direction scanning trajectory. If a pair of candidate boundary positions coincide or are close to each other, that is, their Euclidean distance is at most a distance threshold, such as a value selected from the range of 1 to 10, then they are determined to match. Values ​​in the range of 2-5 obtain the best results. If the mutually overlapping opposite direction scanning trajectories extend along the main coordinate axes x, y of the image, the Euclidean distance metric is simplified to the absolute difference between the coordinate values ​​of the candidate boundary positions to be compared. In one implementation, the boundary indication module 15 then eliminates those candidate boundary positions that do not have matching candidate boundary positions in the overlapping opposite direction scanning trajectories. An example result of this operation is shown in FIG. Figure 12A and Figure 12B shown. Figure 12A Shows the Figure 11AThe results shown are the results of applying the elimination procedure to a set of candidate boundary locations obtained using 36 sets of scanning trajectories with angles differing by 10 degrees. Figure 12B Shows the Figure 11B The results shown are the results of applying the elimination procedure to a set of candidate boundary locations obtained using four sets of scanning trajectories with angles differing by 90 degrees.

[0143] In this example, a further implementation of the boundary indication module 15 also performs a merging operation. That is, if it detects that a pair of candidate boundary positions obtained from a pair of overlapping opposite-direction scanning trajectories do not coincide but are close enough to meet the distance threshold, it replaces the pair of candidate boundary positions with a single candidate boundary position that is centered between the two original candidate boundary positions. The result of this operation is shown in FIG. Figure 13A As shown, where the merging step is applied to Figure 12A The remaining candidate boundary position set is shown. Similarly, Figure 13B The application of the merging step to Figure 12B The results for the remaining set of candidate boundary locations are shown.

[0144] In one variant, the original candidate boundary position is associated with a confidence indicator. The confidence indicator is based on, for example, the fluorescence signal vector F of the scanning trajectory that obtained the original candidate boundary position. L (...). A higher magnitude of the gradient indicates a higher confidence level for the boundary location. In one example, a single candidate boundary location is closer to the original candidate boundary location with the highest confidence level.

[0145] In one embodiment, the boundary indication module 15 performs an operation to eliminate isolated candidate boundary positions. In this embodiment, the boundary indication module 15 identifies a candidate boundary position as an isolated candidate boundary position if there are no other candidate boundary positions within a predetermined distance of the candidate boundary position in the complete set of candidate boundary positions. The predetermined distance may be the same as the predetermined distance in the previously mentioned operation of determining whether a candidate boundary position obtained with a scan trajectory matches a candidate boundary position of an overlapping scan trajectory in the opposite direction. An example result of this elimination operation is shown in FIG. Figure 14 In this example, the elimination operation is applied to Figure 13A The remaining set of candidate boundary locations is shown.

[0146] In one embodiment, boundary indication module 15 performs an operation in which it merges pairs of candidate boundary locations that are within a threshold distance of each other into a single candidate boundary location. In contrast to the previously described merging procedure for merging pairs of candidate boundary locations obtained from overlapping, oppositely oriented scan trajectories, the present merging procedure is applied to any pair of candidate boundary locations in the set of available candidate boundary locations that are within a specified threshold distance of the operation. The specified threshold distance of the operation can also be selected from a range of 2-5. Figure 15 shows the application of the merge operation to Figure 14 Example results for a set of candidate boundary locations are shown. In this example, the merge operation is applied once, but alternatively this operation can be repeated a predetermined number of times or until a stopping criterion is met, such as the number of candidate boundary locations being reduced to a specified threshold number. In practice, it has been found that a single merge step is sufficient. To reduce the computational effort while maintaining acceptable results, the merge operation can be skipped. With this in mind, refer to Figure 13A 、 Figure 13B The described merge operation can also be skipped.

[0147] A boundary is then constructed in the fluorescence image based on the set of candidate boundary positions obtained for each scanning trajectory. The boundary indicates the margin between the tumor and the healthy part of the tissue.

[0148] In one embodiment, the constructed boundary is a master curve interconnecting peripheral candidate boundary locations. The master curve in the fluorescence image is used to indicate the edge of the hyperfluorescent region itself. In one example, where the fluorescence image is obtained from tumor-containing tissue fluoresced with a sufficiently specific tracer, the edge of the hyperfluorescent region may coincide with the edge of the tumor.

[0149] In another embodiment, the boundary is constructed as a secondary curve that encircles the primary curve at interconnected peripheral transition locations and extends beyond the primary curve a distance that depends on the type of tumor present in the tissue. The secondary curve indicates the area that should be treated or resected. Typically, this secondary curve encircles the primary curve at a predetermined distance outside the primary curve, which depends on the type of tumor tissue. In one embodiment, the distance between the secondary curve and the primary curve is locally less than the predetermined distance to avoid intersecting a designated anatomical structure.

[0150] Figure 17 An example of a boundary B constructed in the fluorescence image FI is shown, wherein the boundary construction section 155 directly obtains the candidate boundary position set {pt}a (see FIG. Figure 16 ) constructs the boundary. The boundary construction section 155 thus obtains the filtered candidate boundary position set {pt}a (see Figure 16) Apply the MatLab function boundary(x, y, s) with s = 0.5. This function constructs a boundary that interconnects the peripheral candidate boundary locations. The shape of the constructed boundary is between a shape that closely follows all peripheral candidate boundary locations and a convex hull that surrounds the peripheral candidate boundary locations.

[0151] Example II

[0152] Figure 18A ,..., Figure 18F Example results are shown for fluorescence images obtained from patient tissue in vivo. In this case, the tissue comprises penile squamous cell carcinoma, and the tissue was rendered fluorescent with the tracer Cetuximab-IRDye800CW.

[0153] Figure 18A The filtered candidate boundary position set {pt}a output by the candidate boundary position filtering section 150 is shown superimposed on the original fluorescent image FI. Figure 18B The boundary B constructed directly from the filtered candidate boundary position set {pt}a obtained from the candidate boundary position filtering section 150 is shown superimposed on the fluorescence image.

[0154] Figure 18C The candidate boundary position set {pt}c obtained from the further candidate boundary position matching section 152 is shown superimposed on the fluorescent image.

[0155] Figure 18D Shown from Figure 18C The boundary B constructed from the set of candidate boundary positions {pt}c is shown superimposed on the fluorescence image.

[0156] Figure 18E The candidate boundary position set {pt}n obtained from the candidate boundary position merging section 153 is shown superimposed on the fluorescent image.

[0157] Figure 18F Shown from Figure 18E The boundary B constructed from the set of candidate boundary positions {pt}n is shown superimposed on the fluorescence image.

[0158] Figure 19A ,..., Figure 19F Further example results obtained from fluorescence images obtained ex vivo from intact tissue resected from a patient are shown.

[0159] Figure 19A The filtered candidate boundary position set {pt}a output by the candidate boundary position filtering section 150 is shown superimposed on the original fluorescent image FI. Figure 19BThe boundary B constructed directly from the filtered candidate boundary position set {pt}a obtained from the candidate boundary position filtering section 150 is shown superimposed on the fluorescence image.

[0160] Figure 19C The candidate boundary position set {pt}c obtained from the further candidate boundary position matching section 152 is shown superimposed on the fluorescent image.

[0161] Figure 19D Shown from Figure 19C The boundary B constructed from the set of candidate boundary positions {pt}c is shown superimposed on the fluorescence image.

[0162] Figure 19E The candidate boundary position set {pt}n obtained from the candidate boundary position merging section 153 is shown superimposed on the fluorescent image.

[0163] Figure 19F Shown from Figure 19E The boundary B constructed from the set of candidate boundary positions {pt}n is shown superimposed on the fluorescence image.

[0164] Figure 20A ,..., Figure 20F Further example results obtained from fluorescence images obtained ex vivo from sections of resected tissue are shown.

[0165] Figure 20A The filtered candidate boundary position set {pt}a output by the candidate boundary position filtering section 150 is shown superimposed on the original fluorescent image FI. Figure 20B The boundary B constructed directly from the filtered candidate boundary position set {pt}a obtained from the candidate boundary position filtering section 150 is shown superimposed on the fluorescence image.

[0166] Figure 20C The candidate boundary position set {pt}c obtained from the further candidate boundary position matching section 152 is shown superimposed on the fluorescent image.

[0167] Figure 20D Shown from Figure 20C The boundary B constructed from the set of candidate boundary positions {pt}c is shown superimposed on the fluorescence image.

[0168] Figure 20E The candidate boundary position set {pt}n obtained from the candidate boundary position merging section 153 is shown superimposed on the fluorescent image.

[0169] Figure 20F Shown from Figure 20E The boundary B constructed from the set of candidate boundary positions {pt}n is shown superimposed on the fluorescence image.

[0170] From the example results presented in this example, it can be concluded that the boundary B constructed from the filtered set of candidate boundary locations {pt}a best matches the tumor margin indicated by the pathologist. For example, for a fluorescence image obtained in vivo, such as Figure 18B The constructed boundaries shown provide the best correspondence with the pathologist's analysis.

[0171] Example III

[0172] Figure 21A ,..., Figure 21F Shown are example results obtained from fluorescence images obtained in vivo from patient tissue, including head and neck squamous cell carcinoma, made fluorescent with the tracer Cetuximab-IRDye800CW.

[0173] Figure 21A The filtered candidate boundary position set {pt}a output by the candidate boundary position filtering section 150 is shown superimposed on the original fluorescent image FI. Figure 21B The boundary B constructed directly from the filtered candidate boundary position set {pt}a obtained from the candidate boundary position filtering section 150 is shown superimposed on the fluorescence image.

[0174] Figure 21C The candidate boundary position set {pt}c obtained from the further candidate boundary position matching section 152 is shown superimposed on the fluorescent image.

[0175] Figure 21D Shown from Figure 21C The boundary B constructed from the set of candidate boundary positions {pt}c is shown superimposed on the fluorescence image.

[0176] Figure 21E The candidate boundary position set {pt}n obtained from the candidate boundary position merging section 153 is shown superimposed on the fluorescent image.

[0177] Figure 21F Shown from Figure 21E The boundary B constructed from the set of candidate boundary positions {pt}n is shown superimposed on the fluorescence image.

[0178] Figure 22A ,..., Figure 22F Further example results obtained from fluorescence images obtained ex vivo from intact tissue resected from a patient are shown.

[0179] Figure 22A The filtered candidate boundary position set {pt}a output by the candidate boundary position filtering section 150 is shown superimposed on the original fluorescent image FI. Figure 22BThe boundary B constructed directly from the filtered candidate boundary position set {pt}a obtained from the candidate boundary position filtering section 150 is shown superimposed on the fluorescence image.

[0180] Figure 22C The candidate boundary position set {pt}c obtained from the further candidate boundary position matching section 152 is shown superimposed on the fluorescent image.

[0181] Figure 22D Shown from Figure 22C The boundary B constructed from the set of candidate boundary positions {pt}c is shown superimposed on the fluorescence image.

[0182] Figure 22E The candidate boundary position set {pt}n obtained from the candidate boundary position merging section 153 is shown superimposed on the fluorescent image.

[0183] Figure 22F Shown from Figure 22E The boundary B constructed from the set of candidate boundary positions {pt}n is shown superimposed on the fluorescence image.

[0184] Figure 23A ,..., Figure 23F Further example results obtained from fluorescence images obtained ex vivo from sections of resected tissue are shown.

[0185] Figure 23A The filtered candidate boundary position set {pt}a output by the candidate boundary position filtering section 150 is shown superimposed on the original fluorescent image FI. Figure 23B The boundary B constructed directly from the filtered candidate boundary position set {pt}a obtained from the candidate boundary position filtering section 150 is shown superimposed on the fluorescence image.

[0186] Figure 23C The candidate boundary position set {pt}c obtained from the further candidate boundary position matching section 152 is shown superimposed on the fluorescent image.

[0187] Figure 23D Shown from Figure 23C The boundary B constructed from the set of candidate boundary positions {pt}c is shown superimposed on the fluorescence image.

[0188] Figure 23E The candidate boundary position set {pt}n obtained from the candidate boundary position merging section 153 is shown superimposed on the fluorescent image.

[0189] Figure 23F Shown from Figure 23E The boundary B constructed from the set of candidate boundary positions {pt}n is shown superimposed on the fluorescence image.

[0190] From the example results presented in Example III, it can be concluded that the boundary B constructed from the filtered candidate boundary position set {pt}a best matches the tumor margin indicated by the pathologist. For example, for a fluorescence image obtained in vivo, such as Figure 21B The constructed boundaries shown provide the best correspondence with the pathologist's analysis.

[0191] Example IV

[0192] Figures 24A to 24F Results of fluorescence images obtained from tongue cancer tissue sections fluoresced with the tracer ONM-100 are shown. ONM-100 consists of polymer micelles labeled with indocyanine green (ICG). Chemically, the ONM-100 drug substance comprises a diblock copolymer of polyethylene glycol (PEG) (~113 repeating units) and a poly(methyl methacrylate) derivative covalently bound to functionalized ICG as a fluorophore. ICG content was determined by a qualified method based on its molecular weight of 37.5 ± 12.5 kD.

[0193] Figure 24A The filtered candidate boundary position set {pt}a output by the candidate boundary position filtering section 150 is shown superimposed on the original fluorescent image FI. Figure 24B The boundary B constructed directly from the filtered candidate boundary position set {pt}a obtained from the candidate boundary position filtering section 150 is shown superimposed on the fluorescence image.

[0194] Figure 24C The candidate boundary position set {pt}c obtained from the further candidate boundary position matching section 152 is shown superimposed on the fluorescent image.

[0195] Figure 24D Shown from Figure 24C The boundary B constructed from the set of candidate boundary positions {pt}c is shown superimposed on the fluorescence image.

[0196] Figure 24E The candidate boundary position set {pt}n obtained from the candidate boundary position merging section 153 is shown superimposed on the fluorescent image.

[0197] Figure 24F Shown from Figure 24E The boundary B constructed from the set of candidate boundary positions {pt}n is shown superimposed on the fluorescence image.

Claims

1. An image processing device (1) for processing a fluorescence image obtained from a subject tissue irradiated with excited light, the tissue being photosensitized with a fluorescent agent, the fluorescence image (FI) including a pixel array having respective fluorescence signal values; The image processing device is configured to perform the following operations to indicate a boundary between a target region representing a tissue portion containing a tumor and a reference region outside the target region in the fluorescence image: Obtain each fluorescence signal vector (F) for each scanning trajectory (L) L (...)), where a fluorescence signal vector (F L Each value of (...)) indicates the amplitude of the fluorescence signal in the fluorescence image (FI) at each position (p) of the scanning trajectory (L); where the scan trajectory (L) extends through the tissue region (TA); Evaluation of the vectors for each fluorescence signal (F L (...)) of each corrected contrast-to-noise ratio vector (CNR LM (...)), each contrast-to-noise ratio vector (CNR LM The values ​​of (...)) are calculated for each position (p) of the scanning trajectory (L) as follows: where FB is a reference fluorescence signal value, which is an average value of the reference fluorescence signal values of the fluorescence image, S is a standard deviation of the reference fluorescence signal values, and c is a predetermined constant; where the image processing device is configured to repeat the following steps for each of a plurality of positions along the scan trajectory: Temporarily designate one position (ps) from the plurality of positions along the scan trajectory as a false point indicating the boundary of the target region; Calculate the fluorescence signal vector (F L (...)) corresponding to the position of the scanning trajectory on the first side of the temporarily specified position to obtain a quantity FB0; Calculate the fluorescence signal vector (F L (...)) corresponding to a position of the scanning trajectory on a second side opposite to the first side of the temporarily specified position to obtain a quantity FB1; If FB0 > FB1, it is assumed that the tumor is represented in the first side, and the value FB1 represents the reference fluorescence signal value FB, and the standard deviation (S) is the standard deviation of the fluorescence signal values of the second side; If FB0 < FB1, it is assumed that the tumor is represented in the second side, and the value FB0 represents the reference fluorescence signal value FB, and the standard deviation (S) is the standard deviation of the fluorescence signal values of the first side; If the modified contrast-to-noise ratio vector (CNR LM (...)) if the temporary designated position has a zero crossing point, the temporary designated position is identified as a candidate boundary position; Indicate the boundary based on a set of candidate boundary positions obtained for each scan trajectory.

2. The image processing device (1) according to claim 1, which is configured to indicate the boundary as a main curve (B) interconnecting the peripheral portions of the transition positions.

3. The image processing device (1) according to claim 1, which is configured to indicate the boundary as a secondary curve (B') surrounding the main curve interconnecting the peripheral portions of the transition positions, and the distance by which the secondary curve extends outside the main curve depends on the type of tumor present in the tissue.

4. The image processing device (1) according to claim 3, which is configured to construct the secondary curve (B') in a manner that avoids intersection with a designated anatomical structure.

5. The image processing device (1) according to any one of the preceding claims, wherein the indication F of the fluorescence signal amplitude in the fluorescence image (FI) at the position (p) of the scanning trajectory (L) is L (p) is the average value of the fluorescence signal values ​​of the pixels in the one-dimensional window that includes the position (p) and is transverse to the scanning track direction in the fluorescence image (FI).

6. The image processing device (1) according to claim 5, wherein the average value is a weighted sum of the fluorescence signal values of the pixels within a one-dimensional window according to a Gaussian function having a maximum value at the position (p) of the scan trajectory (L).

7. The image processing device (1) according to claim 1, which is further configured to perform low-pass filtering of the corrected contrast noise ratio vector.

8. The image processing device (1) according to any one of the preceding claims, wherein each scan trajectory includes at least two scan trajectories having mutually different directions.

9. The image processing apparatus (1) according to claim 1, 2 or 3, wherein the fluorescence image (FI) is obtained from a tissue sample obtained from the subject and arranged on a background, and the image processing apparatus (1) is configured to identify a region in the fluorescence image representing the tissue sample as the tissue region, and to identify a background region in the fluorescence image (FI) as the background region.

10. The image processing device (1) according to claim 9 is further configured to determine a first maximum intensity in a tissue area of ​​the fluorescence image (FI), determine a second maximum intensity along the scanning trajectory, and skip further processing steps of the scanning trajectory if the second maximum intensity is less than a predetermined fraction of the first maximum intensity.

11. The image processing device (1) according to any one of the preceding claims, wherein the predetermined constant c is 2.

12. An image processing device (1) according to any one of the preceding claims, comprising a camera (2) for obtaining a fluorescence image (FI) from tissue of a subject to which a fluorescent agent has been administered, and comprising a display device (3) for displaying the tissue and the constructed boundary.

13. The image processing apparatus (1) according to claim 1, configured to reject the candidate boundary position from further processing if the indicator value of the fluorescence intensity is less than a threshold value.

14. The image processing apparatus (1) according to claim 13, wherein the indicator value is a maximum value of fluorescence intensity within a window centered at the candidate boundary position.

15. A medical treatment device (100), comprising: A camera (2) for obtaining a fluorescence image (FI) from tissue of a subject to which a fluorescent agent has been administered, an image processing device (1) according to any one of claims 1 to 14, and A treatment device (5) resects or irradiates the tumor according to the constructed boundaries.

Images