Method for identifying tumor regions

Abstract

The invention relates to a computer-implemented method for marking a region (21) of a tumor (23) in a tissue field image (27), which image shows a tissue region (25) with a tumor (23) and has been obtained by means of light reflected or emitted by the tissue region (25). In the method, the region (21) of the tumor (23) in the tissue field image (27) is marked on the basis of an eigenvalue of the intensity of at least one component of the light reflected or emitted by the tissue region (25). The eigenvalue is determined using the intensity of the at least one component in an image section of the tissue field image (27), which image section corresponds to a tissue segment (36, 36') of the tissue region (25) in which at least one histological information has been obtained.

Description

[0001] This invention relates to a method for marking tumor regions. Specifically, it relates to a computer-implemented method for marking tumor regions in an image of a tissue field, and a method for generating a processed image of a tissue field containing a tumor, in which the tumor region is highlighted. Furthermore, this invention relates to a computer program for marking tumor regions in an image of a provided tissue field, and a non-volatile computer-readable storage medium having instructions for executing the computer program. Additionally, this invention relates to a data processing system that allows marking tumor regions in an image of a provided tissue field. The invention also relates to a medical device for generating a processed image of a tissue field in which the tumor region is depicted in a highlighted manner.

[0002] For example, in surgical procedures aimed at removing brain tumors, treating surgeons face the difficult task of balancing the removal of as much pathological tissue as possible with the preservation of as much functional tissue as possible. To simplify the decision regarding the amount of tissue to be removed, various intraoperative contrast methods are available, among which the accumulation of fluorescent dyes in tumor tissue simplifies the demarcation between tumor and healthy tissue. For instance, US 9,044,142B2 describes an optical surgical system that makes the fluorescence of indocyanine green deposited in tumor cells visible and thus emphasizes the tumor cells. Furthermore, methods for emphasizing tumors using fluorescent dyes are described in US 2017 / 0027446 A1 and US 2010 / 0143258 A1. In particularly important methods used, especially in cases of severe tumors (rapidly growing malignancies) such as glioblastoma, the accumulation of the natural fluorescent metabolite protoporphyrin IX (PpIX) is used to demarcate tumor tissue from healthy brain regions. In this case, PpIX accumulates in the tumor, and if a suitable excitation light and a suitable filter are used in the observation beam path, the PpIX can be identified as a red fluorescent area on a blue background.

[0003] However, despite the use of dyes and fluorescence, determining the boundaries of tumor regions is difficult because, for example, in the peripheral regions of a tumor, tumor cells infiltrate the surrounding healthy tissue, thus reducing the proportion of tumor cells within the tissue. Therefore, in the example using PpIX dye, there exists a purple or pink region where the color transitions from red to blue between the red area marking tumor tissue and the blue area marking healthy tissue. In practice, the tumor boundary is typically located within this mixed region where the color transitions from red to blue. However, it cannot be ruled out that the fluorescence of tumor cells of a particular tumor type may vary between different patients and between different tumors, thus the definition of this type of tumor boundary is subject to a degree of uncertainty.

[0004] Therefore, US 2010 / 0143258 A1 proposed using a threshold of fluorescence intensity, classifying tissue regions with fluorescence intensity above the threshold as tumor tissue and tissue regions with fluorescence intensity below the threshold as healthy tissue, and marking the transition from tumor tissue to healthy tissue. US 2010 / 0143258 A1 does not describe how the threshold is defined.

[0005] Regarding the teachings of US 2010 / 0143258 A1, the objective of this invention is to provide a method for marking tumor regions based on characteristic values ​​of an intensity or time-intensity curve of at least one component of light emitted or reflected by a tissue region, wherein the characteristic values ​​can be determined in an advantageous manner.

[0006] According to claim 1, the aforementioned objective is achieved by a computer-implemented method for marking tumor regions in an image of a tissue field (hereinafter referred to as a tissue field image), and according to claim 12, the aforementioned objective is achieved by a method for generating a processed tissue field image. Furthermore, according to claim 19, the objective is also achieved by a computer program for marking tumor regions in a provided tissue field image; according to claim 20, the objective is also achieved by a non-volatile computer-readable storage medium; according to claim 21, the objective is also achieved by a data processing system; and according to claim 22, the objective is also achieved by a medical device. The dependent claims encompass advantageous configurations of the invention.

[0007] According to the present invention, a computer-implemented method is provided for marking a tumor region in a tissue field image showing a tissue region having a tumor, obtained by means of light reflected or emitted by the tissue region. The device executing this computer-implemented method thereon can, for example, read the tissue field image from memory, receive the tissue field image via a network, or input the tissue field image in any other way. In this case, the tissue field image can be received directly from the imaging device. Within the scope of the invention, the tissue field image should be considered a large-area image, where the large-area image represents 1 cm². 2 or more (e.g., 2cm) 2 5cm 2 (or more) object fields. The tissue field image may optionally be a magnified representation, wherein, however, the magnification is not high enough to resolve cellular structures. Typically, the magnification is in the range of approximately 5x to approximately 40x. In particular, the tissue field image may be an overview image.

[0008] In the computer-implemented method according to the invention, the tumor region is marked based on characteristic values ​​of the intensity or time-intensity curve of at least one component of the light reflected or emitted by the tissue region, the time-intensity curve being, for example, represented by a time constant characterizing the time-intensity curve. In this case, the marking is carried out by means of electronic image processing. The light reflected or emitted by the tissue region may be in the visible spectrum, the infrared spectrum, or the ultraviolet spectrum. In particular, at least one component may be at least one spectral line of fluorescent radiation emitted after excitation by a dye present in the tissue region with some excitation light.

[0009] According to the present invention, feature values ​​are determined based on the intensity or time-intensity curve of at least one constituent part of an image portion corresponding to a tissue segment from which at least one histological information is obtained in a tissue region of a tissue field image. In this case, any information that contributes to the identification of tissue changes and / or cell classification, particularly any information at the cellular level, should be considered histological information. In this case, the image portion of the tissue field image is typically much smaller than the tissue field image itself, and generally corresponds to less than 1% of the image area of ​​the tissue field image, preferably less than 0.5%, and particularly less than 0.1%. For example, histological information may include the proportion of tumor cells, the oxygen content of tumor cells, variables derived from the morphology of tumor cells, etc.

[0010] Therefore, in the computer-implemented method according to the invention, feature values ​​are determined based on an image portion of a tissue field image, which shows a tissue segment in which at least one specific histological information related to the corresponding patient is available. If the histological information obtained based on the tissue segment is a feature value of a portion of a tumor region, such as the edge of the tumor region, then the feature value determined based on the image portion corresponding to this tissue segment is also a feature value of that portion of the tumor region. Therefore, a portion of a tumor region, such as the edge of the region, can be determined very individually for each patient based on the feature values ​​thus determined. Thus, for example, a tumor region can be marked by the edge of the region. In this case, a tumor region can, for example, represent a portion of the tumor where the proportion of tumor cells exceeds a specified value (e.g., a value intended to mark the edge of the tumor). Alternatively, a tumor region can, for example, represent a portion of the tumor where specific tumor properties (e.g., pH value, oxygen content, concentration of H2O2 or other oxygen-containing derivatives, etc.) are higher or lower than a certain limit. In this case, for example, the orientation of the edge of the corresponding region can be inferred by means of histological information.

[0011] For example, histological information can be obtained using rapid sectioning histology. However, alternatively, this histological information can also be specifically included in histological images taken on the image portion of a tissue field image. For example, histological images can be taken using a confocal endoscope, optical coherence tomography (OCT), or a probe with biosensor-type measurement capabilities. For example, in this case, appropriate tissue segments can be selected based on the histological information, wherein the intensity or intensity curve of the light emitted or reflected therefrom is representative of a portion of the tumor region of the corresponding patient, or wherein the intensity or intensity curve of the light reflected or emitted therefrom (which is representative of a portion of the tumor region of the corresponding patient) forms a suitable starting point for calculating the intensity of the reflected or emitted light.

[0012] In a first variant of the computer-implemented method according to the invention, feature values ​​are determined by displaying at least one histological image and providing a selection function for selecting a chosen histological image from these displayed histological images. After the selection function is activated, an actual intensity value or actual time-intensity curve is determined for an image portion showing a tissue segment on which the selected histological image was taken. The determined actual intensity value or the determined actual time-intensity curve is then defined as a feature value of the intensity or time-intensity curve of at least one constituent portion. If, for example, a selected histological image is taken at the edge of a tumor region, the determined actual intensity value or the determined actual time-intensity curve is a feature value of the tumor edge, and therefore the edge of the tumor region can be marked based on the determined actual intensity value or the determined actual time-intensity curve.

[0013] Instead of making a selection based on histological images, there is an option to process at least one piece of histological information contained in at least one histological image and display the processed histological information for each histological image. A selection function is then provided for selecting a specific piece of processed histological information from the displayed processed histological information. After this selection function is activated, an actual intensity value or actual time-intensity curve is determined for the image portion showing the tissue segment, on which a histological image forming the basis of the selected processed histological information is captured, and the actual intensity value or the actual time-intensity curve is defined as a characteristic value of the intensity or time-intensity curve of at least one constituent part. A more objective selection can be made based on processed histological information than based on the histological image itself. Naturally, a selection can be made based on both the histological image and the processed histological information. For example, processed histological information could be a value of the proportion of tumor cells, pH value, oxygen content value, concentration of H2O2 or other oxygen-containing derivatives, etc.

[0014] In this variation, the treating physician or team of treating physicians may, for example, take histological images until a histological image is found that shows characteristic values ​​of a tissue segment that should be considered the edge of a tumor or the edge of a region of a tumor, and then select the corresponding histological image. After selection, for example, the actual intensity value of the image portion of the tissue field image showing this tissue segment is determined and defined as a characteristic value of the intensity of at least one constituent portion. The image region of the tissue field image in which the intensity corresponds to the characteristic value can then be considered the edge of a tumor or the edge of a region of a tumor.

[0015] In a first variation of the computer-implemented method according to the invention, for each captured histological image of a tissue field image corresponding to a tissue segment depicted in a histological image, an actual intensity value or actual time-intensity curve is determined, and the image region in the tissue field image corresponding to the intensity value or actual time-intensity curve of the reflected or emitted light is marked with respect to the respective determined actual intensity value or actual time-intensity curve. If the corresponding actual intensity value or corresponding actual time-intensity curve is used as a feature value, this can indicate to the physician or physician team how large the tumor area should be considered, and this may help to achieve a balance between removing as much tumor tissue as possible and preserving as much healthy tissue as possible. One of the histological images can then be selected based on these considerations. The actual intensity value or actual time-intensity curve of the image portion of the tissue field image corresponding to the tissue segment in the selected histological image can then be defined as a feature value of the intensity or time-intensity curve of at least one constituent portion by activating a selection function.

[0016] In a second variation of the method according to the invention, at least one piece of histological information is quantifiable histological information, such as the tumor cell ratio, and the actual value of the quantifiable histological information determined based on the histological information, along with a specified value of the quantifiable histological information to be used to mark the tumor region (e.g., the tumor margin), is used to determine the characteristic value. For example, the tumor cell ratio can be used as quantifiable histological information, wherein the tumor cell ratio can be considered as the proportion of tumor cells to the total number of cells in a selected tissue segment. However, quantifiable histological information can also be, for example, pH value, oxygen content, concentration of H2O2 or other oxygen-containing derivatives, etc.

[0017] In particular, the determination of actual values ​​of quantifiable histological information can also be performed within the scope of the computer-implemented method according to the invention, for example, based on received histological images. If the value of the quantifiable histological information is the proportion of tumor cells, then determining at least one actual proportion of tumor cells may include the following steps:

[0018] - Identify tumor cells in the received histological images, and

[0019] - Determine the actual proportion of tumor cells in at least one received histological image based on the number of tumor cells identified.

[0020] In this context, the histological image must be helpful in determining the proportion of tumor cells. For example, this histological image could be obtained using a confocal endoscope, an optical coherence tomography (OCT), a probe with biosensor-type measurement capabilities, or magnetic resonance imaging (MRI). However, the histological image could also be an image of a histological section (i.e., an image of a histological slide). For example, the histological image can have a resolution that allows for the identification of individual cells in the image. Preferably, for example, the resolution is even high enough to identify the structure of individual cells, such as the nucleus. The resolution is preferably 10 μm or better, such as 5 μm, 3 μm, 1 μm, or 0.7 μm. For example, tumor cells can then be optionally identified in the histological image based on morphological criteria (e.g., cell structure, nucleus size, etc.) using staining techniques to improve contrast. In this case, the histological image typically shows 1 mm... 2 Or less (e.g., 0.5mm) 2 0.2mm 2 0.1mm 2 Or even fewer) of the object slices, however, tissue field images show 1 cm 2 One or more slices of the object. If the intensity value or time-intensity curve dependency of at least one component of the light reflected or emitted by the tissue is known, the intensity of the light reflected or emitted by the tissue can be used to determine the proportion of tumor cells. This allows the determination of the intensity value or time-intensity curve using the same equipment also used to take histological images.

[0021] Alternatively, the actual proportion of tumor cells can be determined externally, and the determined actual proportion of tumor cells forms the input to the method.

[0022] In this context, standard histological methods can be used to determine the proportion of tumor cells. For example, a suitable method is described in Y. Jiang et al.: "Calibration of fluorescence imaging for tumor surgical margindelineation: multistep registration of fluorescence and histological images", Journal of Medical Imaging 6(2), 025005 (April to June 2019).

[0023] In the first embodiment of the second variation, the feature value can be determined by the following operation:

[0024] - For a tissue field image, determine the actual intensity value or actual time-intensity curve of at least one constituent part of the image corresponding to the tissue segment from which quantifiable histological information is obtained.

[0025] - Calculate the intensity value or time-intensity curve of at least one component at a specified value of quantifiable histological information based on the dependence of the intensity value or time-intensity curve of at least one component on the value of quantifiable histological information, wherein the value of quantifiable histological information is derived from the actual value of quantifiable histological information determined for a tissue segment of a tissue region and the actual intensity value determined for an image portion corresponding to that tissue segment in a tissue field image or the actual time-intensity curve determined for an image portion of a tissue field image, and

[0026] - The calculated value or time-intensity curve of the intensity of at least one component at a specified value of quantifiable histological information is defined as the characteristic value of the intensity or time-intensity curve of at least one component.

[0027] In the second embodiment of the second variation, the feature value can be determined by the following operation:

[0028] - Receive a histological image and determine the actual value of quantifiable histological information of the tissue segment depicted in the received histological image until a tissue segment has been found in which the actual value of quantifiable histological information corresponds to a specified value of quantifiable histological information, the tissue segment being located in the tissue region depicted in the tissue field image.

[0029] - Select the image portion representing the tissue segment in which the actual value of quantifiable histological information corresponds to the specified value of quantifiable histological information;

[0030] - Determine the actual intensity value or actual time-intensity curve of at least one constituent part for a selected image portion; and

[0031] - Define the actual intensity value or actual time intensity curve of the selected image portion as the characteristic value of the intensity or time intensity curve of at least one constituent portion.

[0032] Since the determination of the actual value of quantifiable histological information can be carried out in an automated manner in the second variant, as well as the remaining steps, the determination of feature values ​​based on the captured histological images or multiple captured histological images can be carried out in an automated manner in this variant.

[0033] Therefore, in a second variation of the computer-implemented method according to the invention, histological information, particularly available in the form of histological images, is used to determine, for example, the proportion of tumor cells in a tissue segment of a tissue region depicted in a tissue field image. Furthermore, for an image segment of the tissue field image representing the tissue segment in which the proportion of tumor cells has been determined, the intensity or temporal intensity curve of at least one constituent part is measured. The intensity or temporal intensity curve of the constituent part that can be expected given a specified tumor cell proportion can then be calculated based on the dependence of the intensity or temporal intensity curve of at least one constituent part on the proportion of tumor cells. If such calculation is undesirable or impossible, for example because such dependence of the intensity or temporal intensity curve on the proportion of tumor cells is unknown, there is alternatively an option to determine the actual proportion of tumor cells in a tissue segment of the tissue region until a tissue segment whose actual tumor cell proportion corresponds to the specified tumor cell proportion has been found. For an image portion in the tissue field image depicting this tissue segment, the actual intensity value or actual temporal intensity curve of at least one constituent part is then determined. However, calculating the intensity or temporal intensity curve of the constituent part expected for a specified tumor cell proportion in this case offers the advantage that histological information must be obtained only once and only a single actual tumor cell proportion needs to be determined. This applies not only to the proportion of tumor cells, but also to other quantifiable histological information, such as pH, oxygen content, concentration of H2O2 or other oxygen-containing derivatives.

[0034] In particular, in the computer-implemented method according to the invention, the tissue field image can be a fluorescence image. In this case, the intensity or time-intensity curve of at least one constituent part is the intensity or time-intensity curve of at least one spectral line of fluorescent radiation emitted by the tissue region. The method of using fluorescent dyes to identify tumors is widespread and facilitates a particularly good distinction between tumor cells and healthy cells. Therefore, for example, the intensity or time-intensity curve of fluorescent radiation is a good measure of the proportion of tumor cells in a tissue segment.

[0035] In order to reduce spurious influences from the surrounding environment when determining the intensity or time-intensity curve of at least one component, the computer-implemented method according to the invention can be embodied in such a way as to correct the actual intensity value or time-intensity curve of at least one component based on at least one of the data items included in the following group:

[0036] - Data items that represent the reflective properties of a tissue area, for example, data items that can be used to correct specular reflection in a tissue area.

[0037] - A data item representing the topography of an area of ​​tissue, which allows us to consider the different directions of reflection or emission caused by the topography.

[0038] - This represents a data item representing at least one equipment parameter of the imaging device used to capture tissue field images. This data item can take into account, for example, setting the irradiation intensity, irradiation spectrum, intensity loss due to inserted filters, etc.

[0039] Furthermore, the present invention provides a method for generating a processed tissue field image of a tissue region containing a tumor, wherein the tumor region is marked in the processed tissue field image. The method includes the following steps:

[0040] - Obtain at least one histological piece of information for at least one tissue segment of the tissue region. This at least one histological piece of information may be specifically contained in a histological image taken on the image portion of a tissue field image. For example, the histological image may be taken using an endoscope, such as a confocal endoscope or an endoscope suitable for performing optical coherence tomography.

[0041] - Take tissue field images of the tissue area. In this case, the tissue field image typically represents a 1cm area. 2 or more (e.g., 2cm) 2 5cm 2A large-area image of the object field (or even more). Specifically, this tissue field image can be obtained using a surgical microscope, meaning it can also be taken at magnification. However, in this case, the magnification will not be high enough to allow for the identification of cellular structures. Typically, the magnification ranges from approximately 5x to approximately 40x.

[0042] - The computer-implemented method according to the invention is performed based on at least one obtained histological information and a captured tissue field image, wherein a tissue field image of a region with a labeled tumor is used to form a processed tissue field image.

[0043] The method according to the invention can, for example, be used to process images taken by a surgical microscope to show the boundaries of a tumor to the treating surgeon when the area of ​​the tumor represents the entire tumor, or to show the boundaries of a specific area of ​​the tumor to the treating surgeon, such as the boundaries of certain areas of the tumor in which tumor-specific properties (e.g., pH, oxygen content, concentration of H2O2 or other oxygen-containing derivatives, etc.) are above or below a certain limit.

[0044] In this configuration, the coordinates of the tissue segment from which at least one piece of histological information is obtained can be stored, and a navigation system can be used to orient the imaging device used to capture the tissue field image, such that the tissue segment from which at least one piece of histological information is obtained is imaged in the image portion of the tissue field image. In this way, it can be ensured that the image portion showing the tissue segment is available in the tissue field image of the tissue segment from which at least one piece of histological information is obtained.

[0045] Furthermore, the method may include: specifying a value for quantifiable histological information, such as a value representing the proportion of tumor cells intended to mark the edges of a tumor region; and determining or receiving an actual value for the quantifiable histological information, such as the actual proportion of tumor cells, for at least one tissue segment depicted in an image portion of a tissue field image. To determine the actual value of the quantifiable histological information, a histological image containing the quantifiable histological information may be captured for at least one tissue segment depicted in an image portion of the tissue field image, and the actual value of the quantifiable histological information may be determined based on this histological image. If the actual value of the quantifiable histological information is received, the actual value of the quantifiable histological information is determined externally based on the histological information.

[0046] In particular, the fluorescence image can be captured as a tissue field image, wherein the intensity or time-intensity curve of at least one constituent part is followed by the intensity or time-intensity curve of at least one spectral line of fluorescence radiation emitted by the tissue region.

[0047] Surgical microscopes may include hyperspectral or multispectral sensors for the purpose of capturing tissue field images. Alternatively, or as an alternative, endoscopes may include hyperspectral or multispectral sensors for the purpose of capturing histological images. While conventional image sensors can distinguish only three primary colors, multispectral sensors offer the possibility of distinguishing more than three primary colors, and hyperspectral sensors offer the possibility of distinguishing multiple colors. This makes it possible to detect the intensity or time-intensity profile of at least one constituent part with particularly high precision.

[0048] Furthermore, according to the present invention, a computer program is provided for marking a tumor region in a tissue field image showing a tissue region having a tumor, obtained by means of light reflected or emitted by the tissue region. The computer program includes instructions that, when executed on a computer, cause the computer to mark the tumor region based on characteristic values ​​of the intensity or time-intensity curve of at least one component of the light reflected or emitted by the tissue region. According to the present invention, the computer program includes instructions that cause the computer to determine characteristic values ​​based on the intensity or time-intensity curve of at least one component of an image portion of the tissue field image corresponding to a tissue segment from which at least one histological information of the tissue region is obtained.

[0049] The computer program according to the invention facilitates the execution of the computer-implemented method according to the invention on a computer or any other data processing system. In this case, the computer program can be developed in such a way that it facilitates the development of the computer-implemented method for execution on a computer or any other data processing system.

[0050] Furthermore, the present invention provides a non-volatile computer-readable storage medium storing instructions thereon for marking a tumor region in a tissue field image showing a tissue region having a tumor and obtained by means of light reflected or emitted by the tissue region. The instructions stored on the storage medium include instructions that, when executed on a computer, cause the computer to mark the tumor region based on characteristic values ​​of an intensity or time-intensity curve of at least one component of the light reflected or emitted by the tissue region. Furthermore, the stored instructions include instructions that cause the computer to determine characteristic values ​​based on an intensity or time-intensity curve of at least one component of an image portion of the tissue field image corresponding to a tissue segment from which at least one histological information of the tissue region is obtained.

[0051] The non-volatile computer-readable storage medium according to the invention allows a computer program according to the invention to be loaded onto a computer or any other data processing system and thus allows the computer or data processing system to be configured to perform a computer-implemented method according to the invention. In this case, the instructions stored on the non-volatile computer-readable storage medium may also include instructions that facilitate the development of a computer-implemented method according to the invention.

[0052] According to another aspect of the invention, a data processing system having a processor and at least one memory is provided, wherein the processor is configured to mark the tumor region in a tissue field image showing a tissue region having a tumor, obtained by means of light reflected or emitted by the tissue region, based on instructions of a computer program stored in the memory, and to mark the tumor region based on characteristic values ​​of the intensity or time-intensity curve of at least one component of the light reflected or emitted by the tissue region. In the data processing system according to the invention, the computer program stored in the memory includes instructions that cause the processor to determine characteristic values ​​based on the intensity or time-intensity curve of at least one component of an image portion of the tissue field image corresponding to a tissue segment from which at least one histological information of the tissue region is obtained.

[0053] The data processing system according to the invention, which may take the form of a computer or any other data processing device, facilitates the execution of the computer-implemented method according to the invention. In this case, the data processing system can be developed in such a way that instructions stored in memory facilitate the development of the computer-implemented method according to the invention.

[0054] Furthermore, according to the present invention, a medical device is provided for generating processed tissue field images of a tissue region containing a tumor, in which the tumor region is marked. The medical device according to the present invention includes an image capturing device for capturing tissue field images of a tissue region containing a tumor. The image capturing device may be a camera or image sensor integrated into different equipment. For example, the image capturing device may be an image sensor integrated into a surgical microscope. Furthermore, the medical device includes an interface for receiving at least one piece of histological information and / or for receiving a histological image, the at least one piece of histological information having been obtained for a tissue segment of the tissue region depicted in the image portion of the tissue field image, and the at least one piece of histological information may be obtained based on the histological image. Alternatively, the medical device may also include a histological image capturing device, such as an endoscope, for capturing such histological images. Furthermore, the medical device includes a data processing system according to the present invention. Therefore, the medical device according to the present invention can perform the computer-implemented method according to the present invention, and optionally the development of the computer-implemented method. For example, the histological information in this case may be the proportion of tumor cells, the oxygen content of tumor cells, variables derived from the morphology of tumor cells, etc.

[0055] As image sensors, imaging devices for medical devices may include hyperspectral or multispectral sensors. Alternatively, or as an alternative, histological imaging devices may include hyperspectral or multispectral sensors. This allows for the particularly precise determination of intensity or time-intensity profiles at certain wavelengths.

[0056] Furthermore, the medical device may include an input device for specifying values ​​of quantifiable histological information intended to characterize the edges of a tumor region. For example, this input device could be a keyboard or a touchscreen. However, the input device could also be a voice recognition system, allowing the value of the quantifiable histological information to be input verbally, or a data interface, allowing the specified value of the quantifiable histological information to be transmitted to the medical device.

[0057] A light source with spectral characteristics capable of inducing fluorescence in tissue regions can be used as a light source. In particular, the spectral characteristics can be achieved in this case by an emitter whose spectral maximum is located in the infrared or ultraviolet spectral range. However, this spectral characteristic can also be achieved by a broadband emitter together with a spectral filter.

[0058] Further features, properties, and advantages of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings.

[0059] Figure 1A schematic representation of a medical device for generating processed tissue field images of tissue regions with tumors is shown, in which the edges of the tumor are emphasized.

[0060] Figure 2 A schematic diagram of the structure of a surgical microscope is shown.

[0061] Figure 3 An alternative embodiment of the surgical microscope is shown.

[0062] Figure 4 A flowchart illustrating a first exemplary embodiment of a method for marking the edges of a tumor in a tissue field image is shown.

[0063] Figure 5 A tissue field image showing the tumor is shown schematically, with the tumor's edges highlighted in the tissue field image.

[0064] Figure 6 The histological images, which can be used to determine the proportion of actual tumor cells, are shown in a highly schematic manner.

[0065] Figure 7 A flowchart illustrating a second exemplary embodiment of a method for marking the edges of a tumor in a tissue field image is shown.

[0066] Figure 8 A flowchart illustrating a third exemplary embodiment of a method for marking the edges of a tumor in a tissue field image is shown.

[0067] For illustrative purposes, the invention will be described in detail below based on exemplary embodiments. Figure 1 An exemplary embodiment of a system comprising a surgical microscope 1 as an image-capturing device, an endoscope 3 as a histological image-capturing device, and a computer 5 as a data processing system is shown as a medical device for generating processed tissue field images of tissue regions containing tumors, in which the edges of the tumor are highlighted. In this case, the keyboard 7 of the computer 5 can be used as an input device for specifying values ​​for quantifiable histological information, such as specifying the proportion of tumor cells.

[0068] Figure 1 The endoscope 3 shown includes an optical fiber 9 having a first end 11 and a second end 13. The first end 11 is positioned facing the object of observation 15, which in this exemplary embodiment is a tissue region 25 having a tumor 23 (see [reference]). Figure 5The first end 11 is located in the scanning device 17, which allows the first end 11 to be moved relative to the object being observed 15 in two lateral directions (hereinafter referred to as the x-direction and y-direction). In particular, the scanning device can be implemented using a microelectromechanical system (MEMS). For example, a scanning device using a MEMS is described in US 2016 / 0051131 A1. Refer to this document for the structure of a suitable scanning device.

[0069] The second end 13 of the optical fiber 9 faces the sensor 19, by which the luminous energy incident on the sensor 19 can be captured. The sensor 19 is located in a housing 21, which in this exemplary embodiment is embodied as a separate module but could also be embodied as a handle, and furthermore, the housing houses a light source (not shown) for generating illumination light for illuminating the object 15 and an input coupling device for coupling the illumination light to the second end 13 of the optical fiber 9. In particular, the light source can be a laser light source. However, the light source can also be arranged outside the housing 21 and connected to the housing via a light guide. The output end of the light guide is then located in the housing 21. In this case, the input coupling device input couples the illumination light emitted from the output end of the light guide into the optical fiber 9. The illumination light can be white light (i.e., having a broadband spectrum) or light having a spectrum consisting of one or more narrow band spectral ranges, particularly, for example, spectral lines of one or more narrow band spectral ranges or spectral lines suitable for exciting the fluorescence of a fluorescent dye located in the object 15. For example, the fluorescent metabolite protoporphyrin IX (PpIX) is a suitable fluorescent dye.

[0070] Irradiation light coupled to the second end 13 of the optical fiber 9 is guided through the optical fiber 9 to the first end 11, from which the irradiation light is emitted in the direction of the object being observed 15. Irradiation light reflected by the object being observed 15, or light excited by the irradiation light and emitted by the object being observed 15 (e.g., fluorescence), then enters the first end 11 of the optical fiber 9 and is guided from the first end to the second end 13, from which the irradiation light or fluorescence is emitted in the direction of the sensor 19. Furthermore, focusing optical units may be located at or in front of the ends 11, 13 of the optical fiber 9, and these focusing optical units may be used to focus light onto the surface of the object being observed 15 or onto the sensor 19. In particular, the endoscope 3 may be embodied as a confocal endoscope. Alternatively or as an alternative, the endoscope may also be embodied as an endoscope for performing optical coherence tomography (OCT). For example, confocal microscopy and optical coherence tomography (OCT) are well-known methods and are described in US 2010 / 0157308 A1 and US 9,921,406 B2. Therefore, detailed descriptions of confocal microscopy and OCT are omitted within the scope of this description. Instead, reference is made to US 2010 / 0157308 A1 and US 9,921,406 B2.

[0071] In this exemplary embodiment, the image acquisition using the endoscope 1 is controlled by a computer 5. However, this control can also be implemented using a dedicated control device. In this exemplary embodiment, the computer 5 for control is connected to both the scanning device 17 and the sensor 19. In this exemplary embodiment, the scanning device 17 is controlled by the computer 5 in such a way that it scans the object 15 along a grid of dots. At each scanned grid dot, the object 15 is illuminated with illumination light and the reflected illumination light or the light emitted by the object 15 due to excitation by means of the illumination light is captured. The computer then generates an image based on the reflected illumination light captured at the grid dot or the light emitted by the object captured at the grid dot, the pixel grid of which corresponds to the grid used during scanning. The resolution of the resulting image is typically 10 μm or better, for example, 5 μm, 3 μm, 1 μm, 0.7 μm or even better. In this case, the image typically shows 1 mm. 2 Or less (e.g., 0.5mm) 2 0.2mm 2 0.1mm 2(or even fewer) slices of the object. In this exemplary embodiment, the optical fiber 9, scanning device 17, sensor 19, and computer 5 together form a histological image capturing device, that is, a capturing device for capturing images that help determine histological information, especially quantifiable histological information (e.g., the proportion of tumor cells in the tissue depicted in the image, or the oxygen content, pH value, H2O2, or other oxygen-containing derivative concentrations of the tissue depicted in the image, etc.). For example, tumor cells can then optionally be identified in the histological image based on morphological criteria (e.g., cell structure, nucleus size, etc.) by means of staining techniques for improving contrast.

[0072] Figure 2 As shown, it is possible to Figure 1 A schematic illustration of the possible structure of the surgical microscope 1 used in the setup. Figure 3 Possible alternative structures are shown.

[0073] Figure 2 The surgical microscope 1 shown includes an objective lens 105 as its basic component, which faces the object of observation 15 (that is, the tissue region having a tumor in this exemplary embodiment). This objective lens can be specifically embodied as an achromatic objective lens or an apochromatic objective lens. In this exemplary embodiment, the objective lens 105 consists of two partial lenses that are bonded together to form an achromatic objective lens. The object of observation 15 is arranged in the focal plane of the objective lens 105 such that the object of observation is imaged at infinity through the objective lens 105. In other words, the diverging light beams 107A and 107B emanating from the object of observation 15 are converted into parallel light beams 109A and 109B during their passage through the objective lens 105.

[0074] A magnification converter 111 is arranged on the observer side of the objective lens 105. This magnification converter can be embodied as a zoom system for continuously variable changes in the magnification factor, as in the illustrated embodiment, or as a so-called Galilean converter for step-wise changes in the magnification factor. In a zoom system, for example constructed from a lens combination with three lenses, the two object-side lenses can be displaced to change the magnification factor. However, in practice, a zoom system can also have more than three lenses, such as four or more, in which case the outer lenses can then be arranged in a fixed manner. In a Galilean converter, by contrast, there are multiple fixed lens combinations representing different magnification factors that can be alternately introduced into the beam path. Both the zoom system and the Galilean converter convert the object-side parallel beam into an observer-side parallel beam with different beam diameters. In this exemplary embodiment, the magnification converter 111 is already part of the binocular beam path of the surgical microscope 1, i.e., the magnification converter has lens combinations dedicated to each stereoscopic portion beam path 109A, 109B of the surgical microscope 1. In this exemplary embodiment, the amplification factor is adjusted by means of an amplification factor converter 111 via a motor-driven actuator, which together with the amplification factor converter 111 is part of an amplification factor adjustment unit for adjusting the amplification factor.

[0075] Magnification converter 111 is arranged adjacent to interfaces 113A and 113B on the observer side, via which external devices can be connected to the surgical microscope 1, and in this exemplary embodiment, the interface arrangement includes beam splitter prisms 115A and 115B. However, in principle, other types of beam splitters, such as partial transmission mirrors, can also be used. In this exemplary embodiment, interfaces 113A and 113B are used for output coupling of the beam from the beam path (beam splitter prism 115B) of the surgical microscope 1 and input coupling of the beam into the beam path (beam splitter prism 115A) of the surgical microscope 1.

[0076] In this exemplary embodiment, a beam splitter prism 115A in a partial beam path 109A is used to mirror information or data onto the partial beam path 109A of the surgical microscope 1 for the observer via a display 137 (e.g., a digital mirror device (DMD) or LCD display) and associated optical unit 139. For example, marking lines indicating the direction of the edge of a tumor in the observed tissue region can be superimposed on the image obtained by the surgical microscope 1. A camera adapter 119 is arranged at an interface 113B in another partial beam path 109B, to which a camera 103 is secured, the camera being equipped with an electronic image sensor 123, such as a CCD sensor or a CMOS sensor. Electronic images can be captured by the camera 103, and in particular, digital images of the observed object 15 can be captured. In particular, the image sensor used can also be a multispectral sensor or a hyperspectral sensor, which includes multiple spectral channels instead of just three (e.g., red, green, and blue).

[0077] Following the observer on the upper rear of interface 113 is a binocular tube 127. This binocular tube has two tube lenses 129A and 129B that focus corresponding parallel beams 109A and 109B onto an intermediate image plane 131, i.e., image the observed object 15 onto the corresponding intermediate image planes 131A and 131B. Finally, eyepiece lenses 135A and 135B sequentially image the intermediate image located in the intermediate image planes 131A and 131B at infinity, allowing the observer to view the intermediate image with a relaxed eye. Furthermore, the distance between the two partial beams 109A and 109B is increased within the binocular tube by means of a mirror system or prisms 133A and 133B to adapt the distance to the observer's interocular distance. Additionally, image erection is performed by the mirror system or prisms 133A and 133B.

[0078] Furthermore, the surgical microscope 1 is equipped with an illumination device that illuminates the object 15 with illumination light. For this purpose, in this exemplary embodiment, the illumination device has a white light source 141, such as a halogen lamp or a gas discharge lamp. Light emitted from the white light source 141 is guided in the direction of the object 15 via a deflector 143 or a deflector prism to illuminate the field. Additionally, an illumination optics unit 145 is present in the illumination device, which ensures uniform illumination of the entire observed object 15.

[0079] exist Figure 2In the surgical microscope 1 shown, illumination may be affected. For example, a filter can be introduced into the illumination beam path that transmits only a narrow spectral range of a broad spectrum from the white light source 141, for example, the spectral range that can excite fluorescence of a fluorescent dye located in the object of observation 15. To observe fluorescence, filters 137A and 137B can be introduced into the observation portion of the beam path that filter out the spectral range used to excite fluorescence, allowing the fluorescence to be observed. To illuminate the object of observation 15 using only the spectral range of illumination light required to excite fluorescence, there is an option to use a narrowband light source (e.g., a laser source that emits essentially only in the spectral range required to excite fluorescence) instead of using a white light source along with filters. In particular, the illumination apparatus may also include means to facilitate interchangeability between the white light source and the narrowband light source.

[0080] It should be pointed out that, Figure 2 The illumination beam path shown is highly schematic and does not necessarily represent the actual trajectory of the illumination beam. In principle, the illumination beam path can be designed as so-called oblique illumination, which is consistent with... Figure 2 The schematic illustration in the diagram is closest to the real thing. In such oblique illumination, the beam path extends at a relatively large angle (6° or more) relative to the optical axis of objective lens 5, and as shown in the diagram... Figure 2 As shown, the illumination beam path can extend entirely outside the objective lens. However, alternatively, there is the possibility of allowing the illumination beam path to extend through the marginal region of the objective lens 105, thus enabling tilted illumination. Another possibility for the arrangement of the illumination beam path is so-called 0° illumination, where the illumination beam path extends through the objective lens 105 and is coupled into the objective lens 105 between two partial beam paths 109A, 109B along the optical axis of the objective lens 105 in the direction of the observed object 15. Finally, the illumination beam path can also be designed as so-called coaxial illumination, where there is a first illumination partial beam path and a second illumination partial beam path. These partial beam paths are coupled into the surgical microscope 1 via one or more beam splitters parallel to the optical axes of the observation partial beam paths 109A, 109B, such that the illumination extends coaxially with respect to the two observation partial beam paths.

[0081] exist Figure 2 In the embodiment variant of the surgical microscope 1 shown, the objective lens 105 consists of only one achromatic lens. However, an objective lens system made of multiple lenses, particularly a so-called anamorphic objective lens, can also be used. This objective lens system allows the working distance of the surgical microscope 1 to be changed, i.e., the distance between the object-side focal plane and the vertex of the first object-side lens surface of the objective lens 105, also known as the front focal length. The anamorphic lens also enables the object 15, arranged in the focal plane, to be imaged at infinity, and thus a parallel beam exists on the observer side.

[0082] Figure 3 An example of a digital surgical microscope 148 is shown in a schematic representation. In this surgical microscope, the main objective lens 105, magnification converter 111, and illumination systems 141, 143, 145 are connected to... Figure 2 The surgical microscope 1 with the optical viewing unit shown in the image is identical to the one in the image. The difference lies in... Figure 3 The surgical microscope 148 shown does not include an optical binocular tube. Instead of [the following text is missing] Figure 2 The tube objectives 129A and 129B are from Figure 3 The surgical microscope 148 includes focusing lenses 149A and 149B, which image the binocular observation beam paths 109A and 109B onto digital image sensors 161A and 161B. Here, for example, the digital image sensors 161A and 161B can be CCD sensors or CMOS sensors. Images captured by the image sensors 161A and 161B are transmitted to digital displays 163A and 163B, which can be LED displays, LCD displays, or organic light-emitting diode (OLED) based displays. As in this example, eyepiece lenses 165A and 165B can be assigned to the displays 163A and 163B, which image the images presented on the displays 163A and 163B at infinity, allowing the viewer to view the images with relaxed eyes. Displays 163A and 163B and eyepiece lenses 165A and 165B can be part of a digital binocular tube; however, these displays can also be part of a head-mounted display (HMD), such as a pair of smart glasses. Naturally, images captured by image sensors 161A and 161B can also be transmitted to a monitor. Suitable shutter glasses can be used for three-dimensional viewing of the images depicted on the monitor.

[0083] See below for reference. Figures 4 to 6 A first exemplary embodiment of a method for generating a processed tissue field image 27 is described, which shows a tissue field 25 with a tumor 23. In this case, Figure 4 A flowchart illustrating the method steps implemented on computer 5 is shown. Figure 5 A schematic representation of the processed tissue field image 27 is shown, and Figure 6 A schematic representation of histological image 29 is shown, as used within the scope of generating processed tissue field image 27.

[0084] In the processed tissue field image 27 of this exemplary embodiment, the edge of the tumor 23 is marked by a marker line 21 that surrounds the area of ​​tissue region 25 depicted in the tissue field image 27 where the proportion of tumor cells has or exceeds a specified value. Therefore, the marker line 21 can be considered as a line representing the edge of the tumor. Alternatively, the method can also be designed in such a way that the edge demarcates a certain region of the tumor (e.g., a region where the oxygen content of tumor cells does not exceed a certain value).

[0085] Tissue field image 27, processed by the method described below, is captured using a surgical microscope 1 (i.e., using at least one image sensor contained in the surgical microscope 1). At least one histological image 29 is captured using an endoscope 3. The tissue field image 27 is then processed based on the unprocessed tissue field image 27 and the at least one histological image for the purpose of marking the edges of the tumor 23. The tissue field image 27 is a large-area image of the observed object, showing at least 1 cm. 2 Preferably at least 2cm 2 And usually 5cm 2 Or more. In this exemplary embodiment, a fluorescent dye that accumulates in tumor cells but not in healthy tissue cells is used to capture this tissue field image. To make the fluorescence visible, the object of observation is illuminated with light having a tight spectrum suitable for excitation fluorescence. A filter that blocks the excitation radiation is then introduced into the observation beam path of the surgical microscope 1, such that only the fluorescence radiation can pass through the observation beam path, while the reflected excitation light does not. The Blue 400, referred to as Carl ZeissMeditec AG, is used in this embodiment. TM Within the scope of the method, protoporphyrin IX (abbreviated as PpIX) is used as a dye and causes tumor 23 to be represented in tissue field image 27 by a red fluorescent region 31 on a blue background 33. Due to the invasive characteristics of tumor cells, such as in the case of glioblastoma, there is a transitional region 35 in which both tumor cells and healthy tissue cells are present, and this results in this region having a hue representing a mixed color between red and blue, which becomes redder as the proportion of tumor cells in the tissue segment increases.

[0086] When removing tumors, the difficulty for treating surgeons lies in the desire to remove as much tumor tissue as possible to increase the patient's chances of a cure, while simultaneously wanting to preserve healthy tissue, especially healthy brain tissue in the case of brain tumors. Therefore, the common practice is to locate the tumor margin within a transitional region 35, for example, at a location where fluorescence has a certain intensity value. However, because variations in the fluorescence of tumor cells of a particular tumor type cannot be ruled out between different patients and between different tumor types, the definition of this type of tumor margin is plagued by ambiguity. Similar difficulties also arise outside of Blue 400. TM The method uses fluorescent dyes other than those used in the method. Using the method described in this exemplary embodiment, individual intensity values ​​for each patient can be determined, thereby marking the edges of their tumors.

[0087] The method is based on a large-area tissue field image 27 taken by a surgical microscope 1 (i.e., by at least one of the image sensors of the surgical microscope), which typically shows a tissue area of ​​several square centimeters. The tissue field image 27 may also have a moderate magnification, typically between 5x and 40x. Furthermore, within the scope of the method, at least one histological image 29 is taken using an endoscope 3, and in this exemplary embodiment, the proportion of tumor cells (i.e., the proportion of tumor cells 30 to the total number of cells in the tissue segment 36 depicted in the histological image) is determined based on the histological image. In this exemplary embodiment, a computer 5 performs the method based on these images, and the tissue field image 27 is processed by means of the method in such a way that the edges of the tumor are highlighted in the tissue field image. Figure 5 In this context, highlighting is achieved by emphasizing image regions where the intensity of fluorescence radiation has specific characteristic values. These regions typically form marker lines that enclose tissue areas considered to be tumor tissue.21

[0088] In this exemplary embodiment, the determination of feature values ​​and the processing of tissue field images 27 obtained by the surgical microscope 1 are achieved by means of a computer program running on the computer 5. However, instead of running on the computer 5, the computer program can also run on any other data processing system, such as a data processing system integrated into the surgical microscope 1. The steps performed by the method implemented by the computer program are... Figure 4 It is depicted as a flowchart.

[0089] In the first step S1 of the method, computer 5 receives an unprocessed tissue field image 27 from surgical microscope 1 or its image sensor. Furthermore, in this exemplary embodiment, in step S2, computer 5 receives a specified value for the tumor cell proportion, that is, a specified proportion of tumor cells 30 to the total number of cells in the tissue region, where tissue reaching or exceeding this proportion is intended to be considered tumor tissue. Hereinafter, this value is referred to as the specified tumor cell proportion. However, values ​​for different quantifiable histological information can be specified instead of the tumor cell proportion. For example, if a hypoxic region of the tumor is to be marked, that is, a region where the oxygen content of tumor cells does not exceed a certain value, a value for the oxygen content of the tumor cells can be specified, which is intended to represent the limit of the hypoxic region. Computer 5 can receive the tumor cell proportion or optionally different quantifiable histological information values ​​via input from keyboard 7, via voice input, via receiving from a network, via reading from a computer-readable storage medium, etc. However, there is also the option of storing the specified tumor cell proportion value in the computer program itself. However, it is advantageous in this case if the stored specified tumor cell proportion can be changed by inputting, reading, or receiving an alternative specified tumor cell proportion. The specified percentage of tumor cells can be between 5% and 30%. For example, the specified percentage of tumor cells is typically between 5% and 15%, and can be 10%.

[0090] Then, in step S3, the actual tumor cell proportion or, optionally, a different quantifiable histological information actual value is provided for the tissue segment 36 of the tissue region 25 depicted in the image portion of the tissue field image 27. In this exemplary embodiment, this tissue segment 36 is a portion of the tissue region 25 whose histological image 29 has been captured using the endoscope 3. In this exemplary embodiment, the histological image 29 is used to determine the actual tumor cell proportion of the tissue segment depicted in the histological image 29, that is, the proportion of tumor cells actually present in this tissue segment. For example, the actual tumor cell proportion can be determined based on cell morphology. For example, cell structure or nucleus size can be used as a standard to distinguish tumor cells 30 from healthy tissue cells 32. Alternatively, there is an option to determine the tumor cell proportion using fluorescence methods. For example, the number of fluorescent cells can be determined in the histological image 29. Furthermore, there is an option to perform a biopsy and determine the tumor cell proportion using routine rapid section histology, in which the extracted material can also be stained. In principle, the tumor cell proportion of the tissue segment 36 can also be determined prior to surgery, for example, using magnetic resonance imaging. However, the location where the tumor cell proportion has been determined must be positioned such that it lies within the area of ​​tissue field image 27, and must be marked such that it can be located during operation using a navigation system. Alternatively, values ​​for other quantifiable histological information can be determined using the methods described.

[0091] The actual tumor cell proportion can be determined directly, for example, by means of a program module integrated into a computer program, before providing the actual tumor cell proportion in step S3. This program module is designed to distinguish tumor cells 30 from healthy tissue cells 32 in the histological image 29, for example, based on morphological criteria or based on fluorescence signals emitted from tumor cells. Furthermore, this program module is designed to determine the proportion of the identified tumor cells 30 in the total number of cells to be identified in the histological image 29 and provide this proportion as the actual tumor cell proportion. Alternatively, the actual tumor cell proportion can be determined at a relatively long time before providing the actual tumor cell proportion in step S3, for example, if the determination is performed preoperatively as mentioned above.

[0092] Then, in step S4, the actual intensity value of the fluorescence radiation is determined for the image portion of the tissue segment 36 depicted in the histological image 29, which forms the tissue field image 27. If the fluorescent dye is selected such that the fluorescence intensity at a point in the tissue field image 27 is correlated with the proportion of tumor cells at that point, and furthermore, if the fluorescence in the histological image 29 helps in determining the proportion of tumor cells, the determination of the actual tumor cell proportion and the determination of the actual intensity value can be performed directly and continuously using the same fluorescent dye. In principle, in Blue 400...TM These requirements are met because the fluorescence intensity of PpIX is related to the proportion of tumor cells and because PpIX accumulates in tumor cells, it can be used to identify tumor cells in histological image 29.

[0093] Once the actual tumor cell proportion has been provided in step S3 and the actual fluorescence intensity value has been determined in step S4, these two variables are used in step S5 to determine the fluorescence intensity value at the specified tumor cell proportion. In this exemplary embodiment, the fluorescence intensity value at the specified tumor cell proportion is determined based on calculation.

[0094] In the fluorescent dye PpIX used in this exemplary embodiment, the correlation between a change in fluorescence intensity and a change in the proportion of tumor cells is known. That is, the degree to which the fluorescence signal changes when the proportion of tumor cells changes by a certain amount is known. If the value of fluorescence intensity is now known for a certain proportion of tumor cells, the value of fluorescence intensity for other proportions of tumor cells can also be calculated based on this correlation. The actual intensity value for the actual proportion of tumor cells has been determined in this exemplary embodiment. Therefore, the corresponding value of fluorescence intensity can be calculated based on the correlation of a specified proportion of tumor cells. Finally, in step S6, this calculated value of fluorescence intensity is defined as a characteristic value of the fluorescence intensity at which the edge of the tumor should be marked. In this way, if an actual intensity value associated with any actual proportion of tumor cells has been determined, the fluorescence intensity value for a specified proportion of tumor cells (e.g., 10%) can be calculated.

[0095] Finally, in step S7, the edges of tumor 23 are marked in the tissue field image based on feature values, for example by highlighting image regions with fluorescence intensity that have such feature values. Corresponding image regions are then formed. Figure 5 The image shows the marker line 21. The image regions within the marker line 21 correspond to a tumor cell ratio higher than the specified tumor cell ratio, while the image regions outside the boundary correspond to a lower tumor cell ratio. Because the specified tumor cell ratio is selected in such a way that it is intended to mark the edge of the tumor 23, these regions within the marker line 21 represent the tumor 23, and the regions outside the marker line 21 represent the tissue to be preserved when the tumor is removed.

[0096] Since the actual proportion and intensity of tumor cells are determined based on the current tumor 23 in the patient's body, the described method helps to individually determine the edge of the patient's tumor 23.

[0097] The procedure according to the first exemplary embodiment requires a known correlation between a change in fluorescence intensity on one hand and a change in the proportion of tumor cells on the other. However, even if such a correlation is unknown or too complex, the edge of the tumor can be determined based on fluorescence intensity and histological image 29. The corresponding procedure is explained below based on the second exemplary embodiment of the invention, wherein reference is made to... Figure 7 The flowchart depicted in the document.

[0098] In a second exemplary embodiment, in step S11, a tissue field image is received from the surgical microscope 1, such as regarding Figure 4 The steps described in step S1.

[0099] In step S12, a specified proportion of tumor cells is defined to mark the edge of tumor 23. The procedure in step S12 also corresponds to the procedure from the first exemplary embodiment (that is, the procedure from step S2).

[0100] Then, in step S13, the actual proportion of tumor cells is determined for the tissue segment 36 of the tissue region 25 depicted in the tissue field image 27. In this case, the same method as described with respect to step S3 in the first exemplary embodiment can be used in principle to determine the proportion of tumor cells.

[0101] Then, in step S14 of the second exemplary embodiment, a check is performed regarding whether the tumor cell percentage determined in step S13 corresponds to the specified tumor cell percentage based on a comparison between the tumor cell percentage determined in step S13 and the specified tumor cell percentage. According to this exemplary embodiment, if the determined actual tumor cell percentage value falls within a specified tolerance range around the specified tumor cell percentage, for example, within a tolerance range of ±10% or ±5% around the specified tumor cell percentage, then the determined actual tumor cell percentage corresponds to the specified tumor cell percentage. However, the limits of the tolerance range do not necessarily need to be symmetrical with respect to the specified tumor cell percentage. For example, if a 10% tumor cell percentage is specified, depending on the accuracy of the tolerance range, the actual tumor cell percentage may be considered to correspond to the specified tumor cell percentage if it falls within, for example, a range from 9% to 11%, from 9.5% to 10.5%, or from 9% to 10.5%. Different tolerance ranges may be used depending on the tumor type and the patient.

[0102] If, in step S14, the actual tumor cell proportion is determined not to correspond to the specified tumor cell proportion—that is, if the value of the actual tumor cell proportion is not within the tolerance range around the specified tumor cell proportion—the method proceeds to step S15, where a different tissue segment 36' of the tissue region 25 imaged in the tissue field image 27 is selected. The method then returns to step S13, where the actual tumor cell proportion is determined for the new tissue segment 36'. Steps S13, S14, and S15 are performed until a tissue segment 36' has been found whose actual tumor cell proportion was determined in step S14 to correspond to the specified tumor cell proportion—that is, the actual tumor cell proportion is within the tolerance limits around the specified tumor cell proportion. The method then proceeds to step S16.

[0103] In step S16, an image portion of the tissue field image 27 is selected, wherein the actual tumor cell proportion determined for the tissue segment 36' depicted in the tissue field image corresponds to a specified tumor cell proportion, and an actual intensity value of the fluorescence intensity is determined for this selected image portion. Since the actual tumor cell proportion of this image portion corresponds to the specified tumor cell proportion, the determined actual intensity already represents the fluorescence intensity given the tumor cell proportion. Therefore, the second exemplary embodiment does not require calculating the fluorescence intensity for a specified tumor cell proportion.

[0104] In step S17, the actual intensity value determined in step S16 is defined as a characteristic value of the fluorescence intensity marking the edge of the tumor 23. Finally, in step S18, the edge of the tumor 23 is highlighted by means of this characteristic value, as described with respect to step S17 of the first exemplary embodiment.

[0105] Compared to the method in the first exemplary embodiment, the method in the second exemplary embodiment requires more time because, typically, a larger number of histological images are captured in this second exemplary embodiment than in the first exemplary embodiment, requiring the determination of the actual tumor cell proportion in each of the histological images. However, conversely, knowledge about the correlation between fluorescence intensity and tumor cell proportion is not required.

[0106] Similar to the first exemplary embodiment, in the second exemplary embodiment, a value for another quantifiable histological information can be specified instead of the tumor cell proportion. Then, in step S13, the actual value of this other quantifiable histological information is determined instead of the actual tumor cell proportion.

[0107] The following reference has Figure 8The third exemplary embodiment is described using flowcharts of steps S21 to S29 depicted in the diagram. In the third exemplary embodiment, steps S21, S22, and S23 correspond to steps S11, S12, and S13 of the second exemplary embodiment. Furthermore, step S26 is the same as step S15 of the second embodiment, step S27 is the same as step S16 of the second embodiment, step S28 is the same as step S17 of the second embodiment, and step S29 is the same as step S18 of the second embodiment. Therefore, the main difference between the third exemplary embodiment and the second exemplary embodiment lies in the fact that there is no automated check regarding whether the determined actual tumor cell ratio corresponds to the specified tumor cell ratio. Instead, in step S24, the actual tumor cell ratio is displayed on the monitor of computer 5 or any other monitor or display. Optionally, a histological image 29, based on which the displayed actual tumor cell ratio has been determined, may also be displayed on the monitor or display during this process. In this case, when the user believes that a suitable actual tumor cell ratio exists, the user has the option to generate a trigger signal, for example, by pressing a key or by voice input.

[0108] In step S25, the software checks whether a trigger signal has occurred after a predetermined time interval. If not, the method proceeds to step S26, where a different tissue segment 36' of the tissue region 25 depicted in the tissue field image 27 is selected. The method then returns to step S23, where the actual proportion of tumor cells is determined for the new tissue segment 36'. Steps S23, S24, S25, and S26 are performed until a trigger signal is present. Once a trigger signal is available, the method continues with steps S27, S28, and S29.

[0109] In the first modification of the third exemplary embodiment, instead of the histological image 29 or in addition to the histological image 29, a tissue field image 27 with marker lines 21 generated by the actual tumor cell proportion determined in step S13 is also displayed. Therefore, in the modification of the third exemplary embodiment, steps S27 to S29 are performed after step S23 and before step S24, so that the tissue field image 27 with the marker lines 21 can be displayed in step S24.

[0110] In the second modification of the third exemplary embodiment, step S23, which determines the proportion of actual tumor cells, is omitted. Then, in step S24, only the histological image 29 is displayed. Based on the displayed histological image 29, a pathologist can evaluate the histological information contained in the histological image 29 regarding the depicted tissue segment 36. If the pathologist deems, based on the histological information, that the tissue segment 36 represented in the histological image 29 represents the edge of a tumor, they can generate a trigger signal, and the method continues with steps S27 to S29. Otherwise, the method proceeds to step S26, where another tissue segment 36' of the tissue region 25 imaged in the tissue field image 27 is selected, and then the process returns to step S24 to display the histological image 29 of this tissue segment 36'.

[0111] In a third exemplary embodiment and its modifications, there is also an option to initially capture histological images 29 of multiple different tissue segments 36, 36' and / or determine the associated actual tumor cell proportions and then display the histological images 29 and / or the determined actual tumor cell proportions in step S24. In this case, the computer program provides a selection option by means of which one of the histological images 29 or one of the actual tumor cell proportions can be selected. This selection then leads to the generation of a trigger signal that causes steps S27 to S29 to be performed based on the selected histological image 29 or based on the histological image 29 that forms the basis of the selected actual tumor cell proportion. For selection purposes, the computer program may, for example, display a pointer on a monitor that is placed on the histological image or the actual tumor cell proportion. The selection can then be performed by means of a key or by means of voice input. Alternatively, there is an option to provide the displayed actual tumor cell proportion or the displayed histological image with a number or other identifier. The selection and triggering can then be performed by inputting an identifier assigned to the selected actual tumor cell proportion or assigned to the selected histological image.

[0112] Similar to the first and second exemplary embodiments, in the third exemplary embodiment, a value of another quantifiable histological information can be specified instead of the tumor cell proportion. Then, in step S23, the actual value of this other quantifiable histological information is determined instead of the actual tumor cell proportion.

[0113] In exemplary embodiments, fluorescence intensity can be corrected based on a certain standard. For example, there is an option to determine certain tissue properties, such as specular reflection, by illuminating the image with white light to determine the fluorescence intensity in the tissue field image 27. Furthermore, there is an option to determine the morphology of the tissue region 25 and consider its influence on the representation of the fluorescence image. Similarly, there is an option to consider the equipment parameters of the surgical microscope 1 (e.g., the intensity of the irradiation light that excites fluorescence, the irradiation angle, the setting of the magnification converter, the focus setting, the intensity attenuation caused by the inserted filter, etc.) and thus correct the fluorescence intensity in the captured tissue field image 27. All these corrections are used to determine the true fluorescence intensity affected by the aforementioned process so as to help to more accurately determine the characteristic values. For example, a change in focus may lead to a change in the working distance, which in turn affects the fluorescence intensity captured by the image sensor of the surgical microscope 1. The effect of irradiation intensity is immediately apparent, just like the effect of a filter introduced into the beam path. The effect on the fluorescence intensity at each pixel of the sensor also changes when the magnification changes, because the fluorescence intensity of the object slice is distributed among different numbers of pixels under different magnification settings.

[0114] For illustrative purposes, the invention has been described in detail based on exemplary embodiments. However, those skilled in the art will recognize that they may deviate from the exemplary embodiments without departing from the scope of the invention. In the case of fluorescence methods, fluorescent dyes different from PpIX can be used. For example, peptidyl ligands (chlorine toxins) are also suitable, which specifically bind to tumor cells, particularly glioblastoma cells, and can be dyes that fluoresce in the near-infrared. The corresponding method is described in Y. Jiang et al.: "Calibration of fluorescence imaging for tumor surgical margindelineation: multistep registration of fluorescence and histological images", Journal of Medical Imaging 6(2), 025005 (April to June 2019). Furthermore, instead of using fluorescent dyes, tumor tissue can also be identified in different ways. For example, multispectral sensors or hyperspectral sensors can be used instead of conventional image sensors. Such sensors allow for the identification of typical spectral features of tumor tissue. In this case, fluorescence induced by dyes is no longer needed. Instead of fluorescence intensity, the intensity of certain spectral features is determined for a given proportion of tumor cells. Unlike fluorescence based on light emission, spectral features are based on light reflection. Furthermore, in the described exemplary embodiments, there is an option to determine feature values ​​based on intensity-based temporal decay behavior, particularly based on the temporal decay behavior of fluorescence radiation, rather than on intensity. Therefore, the invention is not intended to be limited to the exemplary embodiments, but only to the appended claims.

[0115] List of reference numerals

[0116] 1. Surgical microscope

[0117] 3. Endoscope

[0118] 5. Computers

[0119] 7 Keyboard

[0120] 9 optical fibers

[0121] 11 Input Terminal

[0122] 13 Output terminal

[0123] 15. Observational Objects

[0124] 17. Scanning device

[0125] 19 Sensors

[0126] 21 Marker lines

[0127] 23. Tumors

[0128] 25 Organizational Areas

[0129] 27. Tissue field images

[0130] 29 Histological images

[0131] 30 tumor cells

[0132] 31 Red fluorescent area

[0133] 32 tissue cells

[0134] 33 Blue background

[0135] 35 Transition Zone

[0136] 36, 36' Organizational segment

Claims

1. A computer-implemented method for marking a region (21) of a tumor (23) in a tissue field image (27), wherein cellular structures are not resolved in the tissue field image, and the tissue field image shows a tissue region (25) having a tumor (23), and the tissue field image has been obtained by means of light reflected or emitted by the tissue region (25), wherein, The region (21) of the tumor (23) is marked in the tissue field image (27) based on the intensity value or characteristic value of the time-intensity curve of at least one component of the light reflected or emitted by the tissue region (25). Its features are, The feature value is determined based on the intensity value or time intensity curve of at least one constituent part in the image portion of the tissue field image (27), the image portion corresponding to the tissue segment (36, 36') of the tissue region (25), from which at least one piece of histological information has been obtained, wherein the histological information is information at the cellular level and is used for the identification of tissue changes and / or the classification of cells.

2. The computer-implemented method according to claim 1, characterized in that, The at least one piece of histological information is contained in the histological image (29) taken on the image portion of the tissue field image (27).

3. The computer-implemented method according to claim 2, characterized in that, To determine this characteristic value, - Display at least one histological image (29), and - Provides a selection function for selecting a selected histological image (29) from the displayed histological images (29). After the selection function is activated, an actual intensity value or actual time-intensity curve is determined for the image portion showing the tissue segment (36, 36'), and the actual intensity value or the actual time-intensity curve is set as the characteristic value of the intensity value or time-intensity curve of the at least one constituent portion, for which the histological image (29) has been taken at the tissue segment.

4. The computer-implemented method according to claim 2 or claim 3, characterized in that, To determine this characteristic value, - For at least one histological image (29), process the at least one piece of histological information contained in the histological image (29), - Processed histological information is displayed for each histological image (29), and - Provides a selection function for selecting a selected piece of processed histological information from the displayed processed histological information. After the selection function is activated, an actual intensity value or actual time-intensity curve is determined for the image portion showing the tissue segment (36, 36') on which the histological image (29) on which the selected processed histological information is based has been captured, and the actual intensity value or the actual time-intensity curve is set as the characteristic value of the intensity value or time-intensity curve of the at least one constituent part.

5. The computer-implemented method according to claim 4, characterized in that, For each captured histological image (29), the actual intensity value or the actual time intensity curve is determined for the image portion of the tissue field image (27) corresponding to the tissue segment (36, 36') depicted in the histological image (29), and the image region in the tissue field image (27) is marked, wherein the intensity value or time intensity curve of the light reflected or emitted in the image region corresponds to the respective determined actual intensity value or the respective determined actual time intensity curve.

6. The computer-implemented method according to claim 1, characterized in that, The at least one piece of histological information is quantifiable histological information, and the feature value is determined using the actual value of the quantifiable histological information and the specified value of the quantifiable histological information for the region (21) of the tumor (23) to be marked.

7. The computer-implemented method according to claim 6, characterized in that, The quantifiable histological information is the proportion of tumor cells, the actual value of which is the actual proportion of tumor cells, and this actual proportion of tumor cells is determined based on the received histological image (29) in the following way: - Identify these tumor cells in the received histological image (29), and - Determine the actual proportion of tumor cells in at least one of the received histological images (29) based on the number of identified tumor cells (30).

8. The computer-implemented method according to claim 6 or claim 7, characterized in that, This eigenvalue is determined in the following way: - For the image portion of the tissue field image (27) corresponding to the tissue segment (36, 36'), determine the actual intensity value or actual time intensity curve of the at least one constituent part, so as to obtain the actual value of the quantifiable histological information for the tissue segment; - Based on the correlation between the intensity value or time-intensity curve of the at least one component and the value of quantifiable histological information, calculate the intensity value or time-intensity curve of the at least one component at a specified value of the quantifiable histological information, wherein the value of the quantifiable histological information is derived from the actual value of the quantifiable histological information determined for the tissue segment (36, 36') and the actual intensity value determined for the image portion of the tissue field image (27) corresponding to the tissue segment (36, 36') or the actual time-intensity curve determined for the image portion of the tissue field image (27) corresponding to the tissue segment (36, 36'), and - The calculated value of the intensity or time-intensity curve of at least one component at a specified value of the quantifiable histological information is set as the characteristic value of the intensity or time-intensity curve of the at least one component.

9. The computer-implemented method according to claim 6 or claim 7, characterized in that, This eigenvalue is determined in the following way: - Receive a histological image (29) and determine the actual value of quantifiable histological information of the tissue segment (36, 36') depicted in the received histological image (29) until a tissue segment (36') is found in which the actual value of the quantifiable histological information corresponds to a specified value of the quantifiable histological information, wherein the tissue segment (36, 36') is located in the tissue region (25) depicted in the tissue field image; - Select an image portion, which represents the tissue segment (36') in which the actual value of the quantifiable histological information corresponds to a specified value of the quantifiable histological information. - Determine the actual intensity value or actual time-intensity curve of at least one constituent part for a selected image portion; and - Set the actual intensity value or actual time intensity curve of the selected image portion to the intensity value or characteristic value of the time intensity curve of the at least one constituent portion.

10. The computer-implemented method according to claim 1, characterized in that, The tissue field image (27) is a fluorescence image and the intensity or time-intensity curve of the at least one constituent part is the intensity or time-intensity curve of at least one spectral line of the fluorescence radiation emitted by the tissue region.

11. The computer-implemented method according to claim 1, characterized in that, The correction of the actual intensity value or actual time intensity curve of the at least one component is performed based on at least one of the data items included in the following group: data item representing the reflectivity of the tissue region, data item representing the morphology of the tissue region, and data item representing at least one equipment parameter of the imaging device used to capture the tissue field image (27).

12. A method for generating a processed tissue field image (27) of a tissue region (25) having a tumor (23), wherein the region (21) of the tumor (23) is marked in the processed tissue field image, characterized by the following steps: - Obtain at least one histological information of at least one tissue segment (36, 36') of the tissue region (25); - Take a tissue field image (27) of the tissue region (25); and -Based on the obtained histological information and the captured tissue field images (27), perform the computer-implemented method as described in any one of claims 1 to 9, wherein the tissue field image (27) of the region (21) having the labeled tumor (23) forms the processed tissue field image (27).

13. The method according to claim 12, characterized in that, The fluorescence image is captured as a tissue field image (27), wherein the intensity value or time-intensity curve of the at least one constituent part is the intensity value or time-intensity curve of at least one spectral line of the fluorescence radiation emitted by the tissue region (25).

14. The method according to claim 12 or claim 13, characterized in that, Take a histological image containing at least one piece of histological information to obtain the at least one piece of histological information.

15. The method according to claim 12, characterized in that, The coordinates of the tissue segment (36, 36') from which at least one histological information is obtained are stored, and the imaging device for capturing the tissue field image (27) is oriented by means of a navigation system, so that the tissue segment (36, 36') from which at least one histological information is obtained is imaged in the image portion of the tissue field image (27).

16. The method according to claim 12, characterized in that, The tissue field image (27) was taken with the aid of a surgical microscope (1, 148).

17. The method according to claim 12, characterized in that, The at least one piece of histological information is contained in the histological image (29) taken on the image portion of the tissue field image (27), which was taken with the aid of an endoscope (3).

18. The method according to claim 12, characterized in that, The actual intensity value used to determine the characteristic value or the actual time intensity curve used to determine the characteristic value is determined by means of an endoscope (3).

19. The method according to claim 16, characterized in that, The surgical microscope (1,148) includes a hyperspectral sensor or a multispectral sensor.

20. The method according to claim 17 or 18, characterized in that, The endoscope (3) includes a hyperspectral sensor or a multispectral sensor.

21. A computer program product for marking a region (21) of a tumor (23) in a tissue field image (27), wherein cellular structures are not resolved in the tissue field image and the tissue field image shows a tissue region (25) having a tumor (23), and the tissue field image has been obtained by means of light reflected or emitted by the tissue region (25), the computer program product comprising instructions that, when executed on a computer (5), cause the computer (5) to mark the region (21) of the tumor (23) based on an intensity value or a characteristic value of a time-intensity curve of at least one component of the light reflected or emitted by the tissue region (25). Its features are, The instructions also cause the computer (5) to determine the feature value based on the intensity value or time intensity curve of the at least one constituent part in the image portion of the tissue field image (27) corresponding to the tissue segment (36, 36') from which at least one histological information is obtained, wherein the histological information is information at the cellular level and is used for the identification of tissue changes and / or the classification of cells.

22. A non-volatile computer-readable storage medium having stored thereon instructions for marking a region (21) of a tumor (23) in a tissue field image (27), in which cellular structures are not resolved and the tissue field image shows a tissue region (25) having a tumor (23), and the tissue field image has been obtained by means of light reflected or emitted by the tissue region (25), the non-volatile computer-readable storage medium comprising instructions that, when executed on a computer (5), cause the computer (5) to mark the region (21) of the tumor (23) based on an intensity value or a characteristic value of a time-intensity curve of at least one component of the light reflected or emitted by the tissue region (25). Its features are, The instructions also cause the computer (5) to determine the feature value based on the intensity value or time intensity curve of the at least one constituent part in the image portion of the tissue field image (27) corresponding to the tissue segment (36, 36') from which at least one histological information is obtained, wherein the histological information is information at the cellular level and is used for the identification of tissue changes and / or the classification of cells.

23. A data processing system having a processor and at least one memory, wherein, The processor is designed to mark the region (21) of the tumor (23) in a tissue field image (27) based on instructions of a computer program stored in the memory, in which cellular structures are not distinguished and the tissue field image shows a tissue region (25) with the tumor (23), and the tissue field image has been obtained by means of light reflected or emitted by the tissue region (25), and the region (21) of the tumor (23) is marked based on the intensity value or characteristic value of the time-intensity curve of at least one component of the light reflected or emitted by the tissue region (25). The feature is that the computer program stored in the memory includes instructions that cause the processor to determine the feature value based on the intensity value or time intensity curve of the at least one constituent part in the image portion of the tissue field image (27) corresponding to the tissue segment (36, 36') from which at least one histological information is obtained, wherein the histological information is information at the cellular level and is used for the identification of tissue changes and / or the classification of cells.

24. A medical device for generating a processed tissue field image (27) of a tissue region (25) having a tumor (23), wherein a region (21) of the tumor (23) is marked in the processed tissue field image, characterized in that, Given: - An image capturing device for capturing tissue field images (27) of a tissue region (25) containing a tumor (23); - A histological image capturing device for capturing a histological image (29) or an interface for receiving at least one piece of histological information and / or for receiving at least one histological image (29), wherein the at least one piece of histological information has been determined for the tissue segment (36, 36') of the tissue region (25) depicted in the image portion of the tissue field image (27); and - The data processing system as described in claim 23.

25. The medical device as described in claim 24, characterized in that, The medical device includes a light source having spectral properties that can induce fluorescence in the tissue region (25).

Images