Systems and methods for training and applying machine learning algorithms for microscopy images

Abstract

The present invention relates to a system (150) for training a machine learning algorithm (160), substantially comprising one or more processors (152) and one or more storage devices (154), configured to receive images (120) depicting tissue, and to tune the machine learning algorithm (160) based on training data (142) such that the machine learning algorithm generates instruction data relating to at least a portion of the tissue depicted in the images (142) and indicative of an action to be taken on the tissue, to provide a trained machine learning algorithm (164). The present invention further relates to a system and corresponding method for applying such a machine learning algorithm.

Description

[Technical field]

[0001] The present invention relates essentially to systems and methods for training machine learning algorithms based on images, e.g., fluorescent images from a surgical microscope, to provide trained machine learning algorithms, and systems and methods for applying said machine learning algorithms. [Background technology]

[0002] In surgical microscopy, such as for tumor surgery, a surgeon can view a surgical site or a patient by using a surgical microscope. A decision must typically be made by the surgeon as to which tissue or portion of a tissue, such as tumor tissue, should be excised or removed. Summary of the Invention [Means for solving the problem]

[0003] In view of the above situation, there is a need for improvements in providing assistance as to which tissue parts should or should not be removed. According to embodiments of the present invention, a system and a method for training a machine learning algorithm, for example on the basis of images from a surgical microscope, a trained machine learning algorithm and a system and a method for applying said machine learning algorithm are proposed having the features of the respective independent claims. Advantageous further developments form the subject matter of the respective dependent claims and the following description.

[0004] One embodiment of the present invention relates to a system for training a machine learning algorithm, e.g. an artificial neural network, comprising one or more processors and one or more storage devices. The system is configured to receive training data including images showing tissue, where the images can be captured using a surgical microscope or from radiography or otherwise. The system is further configured to determine an adjusted (trained) machine learning algorithm based on the training data, i.e. the machine learning algorithm is configured to train the machine learning algorithm to generate instruction data for at least a portion of the tissue shown in the images, e.g. indicative of an action to be performed on the tissue during surgery. Such instruction data can be or include a resection border, etc. The system is further configured to provide the trained machine learning algorithm, where the machine learning algorithm is configured to provide the instruction data, e.g. in intraoperative use of a surgical microscope.

[0005] In surgery, tumors and other harmful tissues can be resected or removed from the patient or the patient's brain, non-harmful tissues, etc. As mentioned above, the surgeon must determine (during surgery) exactly which tissues or parts should or should not be resected. In particular, in neurosurgical oncology, there are additional limitations related to the side effects of neural tissue resection, since removal of viable brain tissue, even if the tissue is malignant, can cause impairment of brain function, which may be more significant than the damage caused by the tumor. Thus, in neurosurgical oncology, the success of surgical intervention is not always a complete resection of the diseased (harmful) tissue, but rather a delicate balance is preferably struck between tumor resection and preservation of neural tissue. It has also been found that sometimes tumors are invasive and scattered outside the solid tumor components (core), and such tumors can be difficult to detect during surgery.

[0006] Typically, optimal surgical intervention planning is empirical, relying on the surgeon's experience to evaluate all preoperative and intraoperative data. These data include, for example, MRI tumor depiction, intraoperative visual evaluation (white light imaging), intraoperative tumor depiction (e.g. based on fluorescent imaging), and preoperative and intraoperative determination of functional areas. Collective evaluation of all data is a very challenging task for the surgeon, especially during surgery. The above-mentioned system for training a machine learning algorithm provides a method for the machine learning algorithm to automatically indicate to the surgeon which tissue or tissue part should or should not be removed (by instruction data). The training data used for training can be collected, for example, from many surgeries or other situations or cases, as described below.

[0007] According to another embodiment of the present invention, the training data further comprises patient data, which includes information about the patient that correlates with the tissue shown in the image. The patient data preferably includes at least one of the following parameters about the patient: age, sex, type and dose of tumor treatment, life expectancy, blood pressure, cholesterol level, tumor recurrence rate, at least partial loss of one or more cognitive functions, and other measurable data in the patient's health file. This allows supervised training or learning of machine learning algorithms that take into account different characteristics and measures, resulting in different decisions about whether harmful tissue should be removed or not.

[0008] In another embodiment of the invention, the machine learning algorithm is adapted to generate prescription data such that the prescription data is based at least in part on at least a portion of the patient data, which is provided as input data to the trained machine learning algorithm for application. In this way, the prescriptions provided to the surgeon are individualized to the current patient, with the prescriptions being more or less individualized depending on the number of different types of patient data used.

[0009] According to another embodiment of the invention, the training data further includes annotations for the images, the annotations indicating at least one of a class or type of tissue shown in the images and an action to be performed or performed on the tissue.

[0010] Such annotations indicating the type of tissue part or part shown in the corresponding image can be labels received, for example, from histopathological and / or other processes after resection, such as "tumor or not tumor" labels (i.e. classes like "tumor", "not tumor"). Annotations indicating the action to be performed or performed on the tissue can be labels like "tumor but not resect", "remove only part of the part", etc. According to a further embodiment, the machine learning algorithm is based on classification. This allows training of a type classification of machine learning algorithms, in particular based on at least a part of the annotations. Training here requires the annotations on images as training data. In addition, with the patient data and the action to be performed or performed as further training data, such a type classification of machine learning algorithms can determine, after training, whether or not a certain part of tissue should be resected in real-time imaging. It should be noted that the type classification of machine learning algorithms does not require calculation of the patient's life expectancy after surgery, since the tissue is image-classified according to the "correlation" of various signals of the image with the provided annotations. Such signals in an image can include, for example, pixel color, spectrum, fluorescence intensity, glossiness, etc. The learned correlations can then be used to predict, for example, different brain tissue classes.

[0011] According to another embodiment of the invention, the machine learning algorithm is based on regression. This allows for a regression type training of the machine learning algorithm. The regression here can preferably be based at least in part on the annotations on the images. In other words, another (or additional) approach is a regression type machine learning algorithm, which can be trained by finding "correlations" between images, i.e. features in the images such as pixel color, spectrum, fluorescence intensity, glossiness, etc., and life expectancy (survival time) after surgery. Thus, such a regression type machine learning algorithm, after training, allows in real-time imaging to decide whether a certain part of tissue should be resected, which corresponds in particular to calculating the life expectancy alone after surgery. It should be noted that the multiple calculations of life expectancy are preferably based on the selection of multiple proposed resection boundaries.

[0012] The difference between regression and classification, among other things, is that regression looks at the outcome of the entire surgery, while classification simply classifies different areas of tissue individually, and therefore, in this case, regression does not require annotation, at least in theory.

[0013] According to another embodiment of the invention, the trained machine learning algorithm is determined on the basis of unsupervised learning, i.e. the training is unsupervised training or unsupervised learning. Such a training method does not require the desired output data (annotations or actions to be performed) as training data, but only the images, e.g. from a surgical microscope, acquired during surgery.

[0014] According to another embodiment of the invention, the training data further comprises at least one of radiological images or scans, ultrasound images, endoscopic images, neuromonitoring information. Each of these images may preferably correspond to an image showing said tissue, for example in terms of a field of view. Such radiological images or scans may be obtained, for example, from or by magnetic resonance imaging (MRI), computed tomography (CT) or the like. This further improves the training of machine learning algorithms by providing additional information in the training data. For example, a particular structure in particular in a tissue may represent a harmful tissue or tissue part or a non-harmful tissue or tissue part, but cannot be clearly observed as such in the (microscopic) image. In particular, said radiological images are obtained from radiological scans, having the same or similar field of view as the corresponding (microscopic) image. These radiological images as well as the other types of images and information mentioned above can be annotated, preferably in the same manner as mentioned for general images, thereby increasing the number of images and information and thus the type of information to be used as training data.

[0015] According to another embodiment of the invention, the prescription data includes boundaries indicating an area of ​​the tissue within which an action on the tissue should be performed, so that the prescription data can state that all tissue within a certain boundary should be excised, which is then also made by the surgeon, thereby increasing confidence in the decisions provided to and made by the surgeon as to what or what not to excise.

[0016] According to another embodiment of the invention, the boundary is determined from a plurality of boundaries with different values ​​of a parameter related to the tissue or related to the patient from which the tissue originates. In this case, the boundary is determined such that the value of the parameter is maximized. This is particularly the case for regression type machine learning algorithms. For example, the parameter can be the patient's life expectancy after the current operation. In this case, from a plurality of boundaries, the boundary corresponding to the longest life expectancy is determined. Another (or additional) parameter can be the quality of life after the operation, for example indicating whether the patient can still use all his neurological skills, etc. It should be noted that this may mean that not all harmful tissues are within the boundary.

[0017] According to another embodiment of the invention, the instruction data includes different indicators for different actions to be performed on the tissue. The different indicators may be, for example, different colors and / or different shades. This is particularly true for classification types of machine learning algorithms. This allows the surgeon to, for example, ascertain what type of tumor is present and what to do.

[0018] According to another embodiment of the invention, a machine learning algorithm is adapted to generate said instructional data overlaid on said microscopic image, for example, this allows a real-time image to be overlaid on the boundary to directly indicate which tissue should or should not be excised.

[0019] A further embodiment of the invention relates to a computer-implemented method for training a machine learning algorithm, the method including receiving training data including images showing tissue. The method further includes adjusting the machine learning algorithm based on the training data such that the machine learning algorithm generates the instructional data for at least a portion of the tissue shown in the images and indicative of an action to be performed on the tissue. The training data may further include the patient data and / or other types of images as described above. The method further includes providing the trained machine learning algorithm to a user or a system, e.g., for application.

[0020] Another embodiment of the invention relates to a trained machine learning algorithm that is trained by receiving training data including images showing tissue, and adjusting the machine learning algorithm based on the training data such that the machine learning algorithm generates instructional data for at least a portion of the tissue shown in the images and indicative of an action to be taken on the tissue. The training data may further include the patient data and / or other types of images as described above.

[0021] Another embodiment of the present invention relates to a system for providing instruction data, e.g., in the form of a boundary or a surrounding boundary, for tissue shown in a microscopic image during surgery, e.g., comprising one or more processors and one or more storage devices. The system is configured to receive input data including a microscopic image from a surgical microscope acquired during surgery. The microscopic image shows tissue of a patient. The input data may further include other of preoperative radiological images or scans and / or other types of images as described above. The input data may further include patient data, such as relevant data from a patient's medical record. The system is further configured to generate instruction data for at least a portion of the tissue shown in the microscopic image by applying a machine learning algorithm. The instruction data indicates an action to be performed on the tissue. The system is further configured to provide output data including instruction data for the microscopic image. For example, the instruction data (e.g., a boundary or a resection boundary) may be overlaid on the microscopic image.

[0022] According to another embodiment of the invention, the input further comprises patient data, where the prescription data is determined at least in part based on at least a portion of said patient data, where the patient data can be provided to the system prior to commencing surgery.

[0023] Another embodiment of the invention relates to a surgical microscope system including a surgical microscope, an image sensor, and a system for providing indication data according to the above-mentioned embodiments.

[0024] A further embodiment of the invention relates to a computer-implemented method for providing instructional data relating to tissue shown in a microscopic image. The method includes receiving input data including a microscopic image of a patient's tissue from a surgical microscope acquired during surgery. The method further includes generating instructional data for at least a portion of the tissue shown in the microscopic image by applying a machine learning algorithm, the instructional data indicating an action to be performed on the tissue. The input data may further include other of pre-operative radiological images or scans and / or other types of images as described above. The input data may further include patient data, such as relevant data from a patient's medical record. The method further includes providing output data including instructional data for the microscopic image.

[0025] A further embodiment of the invention relates to a method for providing images and instruction data to a user, e.g., a surgeon, using a surgical microscope. The method includes illuminating tissue of a patient and capturing a microscopic image of the tissue, the image being indicative of the tissue. The method further includes generating instruction data for at least a portion of the tissue shown in the microscopic image by applying a machine learning algorithm, the instruction data being indicative of an action to be performed on the tissue. The method further includes providing the microscopic image and the instruction data to a user of the surgical microscope. The method further includes importing data, i.e., patient data, such as pre-operative radiological images or scans and / or other types of images as described above and / or relevant data from a patient's health record.

[0026] For advantages and further embodiments of the method, reference is also made to the correspondingly applied system description herein.

[0027] Another embodiment of the invention relates to a computer program comprising a program code for performing the above method, when the computer program is executed on a processor.

[0028] Further advantages and embodiments of the invention will become apparent from the description and accompanying drawings.

[0029] It should be noted that the features mentioned above and those further described below can be used not only in the respective indicated combinations, but also in other combinations or alone, without departing from the scope of the present invention. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 illustrates a schematic of a system for training a machine learning algorithm, according to an embodiment of the present invention. [Diagram 2] FIG. 2 illustrates a schematic diagram of a system for providing indication data according to another embodiment of the present invention. [Diagram 3] FIG. 1 illustrates a schematic diagram of how annotations are created that are used to train machine learning algorithms, according to an embodiment of the present invention. [Figure 4a] FIG. 2 illustrates a schematic diagram of indicative data generated by application of a machine learning algorithm, according to an embodiment of the present invention. [Figure 4b] FIG. 2 illustrates a schematic diagram of indicative data generated by application of a machine learning algorithm, according to an embodiment of the present invention. [Diagram 5] FIG. 1 illustrates generally a method for training a machine learning algorithm according to an embodiment of the present invention. [Figure 6] FIG. 4 illustrates a schematic diagram of a method for correcting a microscopic image according to another embodiment of the present invention. [Figure 7] FIG. 4 illustrates generally a method for providing an image to a user according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] 1, a system 150 for training a machine learning algorithm, according to one embodiment of the present invention, is illustrated generally. System 150 includes one or more processors 152 and one or more storage devices 154. For example, system 150 may be a computer or other server system.

[0032] The system 150 is configured to receive training data 142, including images, e.g., microscopic images 120 from a surgical microscope 100, acquired during surgery. The (microscopic) images show tissue, such as a patient's brain, including tumors and / or other harmful tissue. The system 150 is further configured to tune a machine learning algorithm 160 based on the training data 142 such that, when the machine learning algorithm is applied, the machine learning algorithm generates instructional data relating to at least a portion of the tissue shown in the images 120, as will be described in more detail with respect to FIG. 2. The system 150 is further configured to provide a tuned (or trained) machine learning algorithm 164, e.g., for an end use.

[0033] According to another embodiment of the invention, the training data 142 includes annotations 132 for images, e.g., microscopy images 120. The annotations here may include or be, for example, a label indicating whether a portion in the microscopy image 120 is a tumor or not. The label may also indicate whether a portion should be excised or not. In this way, the annotations may indicate the class of the tissue portion shown in the corresponding microscopy image 120 and the action to be or has been performed on the tissue, which may be used for at least one machine learning algorithm of classification and regression type.

[0034] According to another embodiment of the invention, the training data 142 includes patient data 136, which contains information about the patient that correlates to the tissue shown in the image 120. According to one embodiment of the invention, the patient data 136 includes at least one of the following parameters for the patient: age, sex, type and dose of tumor treatment, (post-operative) life expectancy, blood pressure, cholesterol level, etc. (i.e. all measurable data in the patient's health file, as well as data describing the patient's quality of life after surgery, such as tumor recurrence rate and loss of some cognitive abilities (e.g. speech), which may be categorical variables). During statistical analysis, some variables (e.g. sex) may be found as redundant or irrelevant.

[0035] According to another embodiment of the invention, the training data 142 includes radiological images 134, which may be obtained from radiological scans, in particular by finding the same fields of view in the radiological scans that correspond to the microscopic images 120. These radiological images 134 may be annotated in the same manner as the microscopic images 120. Ultrasound images, endoscopic images and neuromonitoring information may also be included in the training data 142.

[0036] In the following, with reference to FIG. 1, it will be explained in more detail how the microscopic images 120, the annotations 132 and the patient data 136 that can be used as training data 142 are obtained.

[0037] During surgery, a surgeon 110 (user) uses a surgical microscope 100 to observe a surgical site, for example a patient 112 or a patient's brain. Said surgical microscope 100 can include illumination optics 102 for visible light and illumination optics 104 for excitation light for exciting fluorophores in the tissue of the patient 112. Alternatively, suitable filters can be used to filter light of the wavelengths required for excitation and emission from the tissue. An image sensor 106, for example a detector or a camera, captures the fluorescence emitted from the illuminated tissue. The image sensor 106 can also capture visible light. Alternatively, a separate image sensor can be used for visible light. In this way, raw data 118 of a microscopic image 120 (in particular a visible light image and / or a fluorescent image or a combination of a visible light image and a fluorescent image) is generated. Such raw data can be processed to obtain a (final) microscopic image 120. Such processing of the raw data can be performed in the image sensor 106 or in another processor included in the surgical microscope 100 or in another external processor (not shown here). The processing of the raw data 118 may include, for example, applying filters, etc. The microscopic images 120 are then stored in a database 122.

[0038] It should be noted that such microscopic images 120 can be formed or captured several times during an operation, in particular before, during and after the resection of harmful tissue such as a tumor. In this way, multiple microscopic images 120 can be captured from a single operation. Similarly, other microscopic images 120 can be captured before, during and after other (different) operations, in particular involving the resection of similar, i.e., the same kind or type of harmful tissue. This allows a large number of microscopic images 120 to be collected and stored in the database 122. A further approach to increase the amount of images and the variety of information is to obtain radiological images 134 from radiological scans, ultrasound images and endoscopic images as described above. Information can also be captured from neuromonitoring, which can also be stored in the database 122.

[0039] In this case, these microscopic images 120 and radiological images 134 can be viewed, for example, on a computing system with a display running the annotation application 130. The surgeon 110 or any other qualified user can view pairs of microscopic images of the same procedure in the annotation application 130 from the microscopic images 120 in the database 122. Such pairs of microscopic images include pre-resection and post-resection images showing the marked tissue portions.

[0040] It should be noted that the microscope image here can preferably include a marked portion. Such (harmful) tissue portions can be marked by a fluorophore or other staining material or marker, typically administered to the patient. A typical fluorophore for marking harmful tissues such as tumors is, but is not limited to, 5-ALA (5-aminolevulinic acid). There are other fluorescent bodies that can be used. In this case, during surgery, the surgical site is illuminated by excitation light of a suitable wavelength to excite the fluorophore or marker to emit fluorescence. The emitted light here can be captured or captured by the surgical microscope or its imager, and can further be displayed, for example, on a display. Fluorescent images can be used here, and the imaging method here is also referred to as fluorescent imaging. In this way, the surgeon can more easily identify tissues that may have to be excised or whether tissues that may have to be excised are still present after excision.

[0041] Such markings may assist the surgeon or user of application 130 to (more easily) distinguish between harmful and non-harmful tissue, although it should be noted that such markings are not required to create an annotation.

[0042] Post-resection images typically contain either non-harmful tissue that was not supposed to be removed during surgery (although it was marked), or harmful tissue that was intentionally left behind because it could not be removed for various reasons (e.g. brain areas responsible for some cognitive process, such as speech).

[0043] An annotation 132 for each such pair of microscopic images 120 can be made, indicating a class of the tissue portion shown in the corresponding microscopic image, such as "tumor" or "not tumor," and further, such annotation 132 can indicate an action to be taken or has been taken on the tissue. For example, the annotation can state "tumor but not resected" (e.g., if the tissue portion is a brain region responsible for some cognitive process, such as speech).

[0044] In a similar or similar manner, the pair of radiographic images 134 and other image pairs can be viewed and annotated, in which case the annotations 132 described above can include annotations to the radiographic images or other images.

[0045] In this case, such annotations 132 and patient data 136 may be stored, preferably along with the microscopic images 120 and / or radiological images 134 and / or other images, in a separate database 140. Other microscopic images 120, other annotations 132 and / or other patient data 136 may also be provided or created at other locations and / or by other personnel, and these images may also be stored in the database 140.

[0046] Such radiological images or scans 134 may be obtained by MRI, CT, etc. Such radiological images or scans typically provide other tissue details than microscopic images, thereby improving the training by adding further information.

[0047] When training the machine learning algorithm 160, the training data 142, preferably including microscopy images 120, annotations 132 and / or patient data 136, as well as radiological images or scans 134 and / or other images or information, may be provided to the system 150 from a database 140.

[0048] In particular, for a classification type machine learning algorithm, the training data 142 may thus include the microscopic images 120 (and radiological images 134) and / or the patient data 136 as input data, and the annotations 132 or parts thereof as desired output data of the machine learning algorithm. The machine learning algorithm provides or generates instruction data regarding the microscopic images 120 of tissue received as input and indicating an action to be performed on said tissue. Such instruction data may in particular have different colors or include other indicators for the performance of different actions (e.g., to excise, not to excise).

[0049] To obtain the indicator, for example, weights of a machine learning algorithm (which may be an artificial neural network) must be adjusted, and for example, the weights can be iteratively adjusted until instruction data for the microscopic image 120 produced by the machine learning algorithm includes the desired action based on the annotation.

[0050] In particular, for a regression type machine learning algorithm, thus, the training data 142 may include the microscopic image 120 (and the radiological image 134) and / or the patient data 136 as input data for the machine learning algorithm. The machine learning algorithm receives as input, with respect to the microscopic image 120 of tissue, and provides or generates instruction data indicating an action to be performed on said tissue. The instruction data may include, in particular, the boundary of the area in which the action is to be performed (e.g. the area to be excised). The boundary here may be the boundary within which the patient's life expectancy is maximized if the tissue is excised.

[0051] To obtain the indicative data, for example, weights of a machine learning algorithm (which may be an artificial neural network) must be adjusted, where for example the weights can be iteratively adjusted until the input data or its features correlate with the life expectancy of the patient from which the microscopic image 120 originates.

[0052] Ultimately, the trained machine learning algorithm 164 can be provided for further application, for example, during surgery.

[0053] In Fig. 2, a system 250 for providing indication data relating to tissue shown in a microscopic image, in particular for the application of a (trained) machine learning algorithm, according to another embodiment of the present invention is shown in a schematic manner. Also shown in Fig. 2 is a surgical microscope 200. The system 250 includes one or more processors 252 and one or more storage devices 254. For example, the system 250 can be a computer or controller, in particular a part of a surgical microscope, for example the surgical microscope 200, or can be integrated into the surgical microscope 200.

[0054] Said surgical microscope 200 may include illumination optics 202 for visible light and illumination optics 204 for excitation light for exciting fluorophores in the tissue of the patient 212. Alternatively, suitable filters may be used to filter the wavelengths of light required for excitation. An image sensor 206, for example a detector or a camera, captures the fluorescence emitted from the illuminated tissue. The image sensor 206 may also capture visible light. Alternatively, a separate image sensor may be used for visible light. In this way, raw data 218 of a microscopic image 220 (in particular a visible light image and / or a fluorescent image or a combination of a visible light image and a fluorescent image) is formed. The raw data here may be processed to obtain a (final) microscopic image 120. Such processing of the raw data may be performed within the image sensor 206 or in a separate processor included in the surgical microscope 200 or in a separate external processor (not shown here). The processing of said raw data 218 may include, for example, applying filters. It should be noted that the surgical microscope 200 may correspond to the surgical microscope 100 of FIG. 1.

[0055] During surgery, a surgeon 210 (user) uses the surgical microscope 200 to observe a surgical site, for example a patient 212 or a patient's brain. The surgical microscope 200 captures microscopic images 220, which may be captured continuously in real time, and the following describes the application of a trained machine learning algorithm 264 to a single microscopic image 220. The trained machine learning algorithm 264 may be applied to each image (frame) of an image sequence or video accordingly.

[0056] The system 250 is configured to receive input data including a microscopic image 220 from the surgical microscope 200, the microscopic image 220 being acquired or captured during surgery and showing the tissues of a patient 212. The microscopic image 220 can be received directly or indirectly from the surgical microscope 200 or its image sensor 206, in particular the image sensor 206 can form raw data 218 which is then processed into a final microscopic image 220. The input data further includes patient data 236 containing information about the patient 212 currently undergoing surgery.

[0057] The system 250 is further configured to generate prescription data 232 for at least a portion of the tissue shown in the microscopic image 220 by applying a machine learning algorithm 264. The machine learning algorithm 264 preferably corresponds to or is the machine learning algorithm 164 trained by the system 150 described with reference to FIG. 1. The prescription data 232 indicates an action to be performed on the tissue and is based at least in part on the patient data 236. The system 250 is further configured to provide output data including the prescription data 232. The prescription data 232 can then be presented to a user, e.g., the surgeon 210, on a display 270 along with and / or overlaid on the microscopic image 220. Note that this also allows other personnel to monitor the prescription data during the procedure.

[0058] Fig. 3 shows a schematic diagram of a method for creating annotations for use in training a machine learning algorithm, for example as described in relation to Fig. 1. In an annotation application 330, which may correspond to the annotation application 130 of Fig. 1, for example a microscopic image 320a and a microscopic image 320b are shown, for example on a display. Both microscopic images 320a and 320b are obtained from a surgical microscope as described in relation to Fig. 1, but the microscopic image 320a is obtained before the resection (or surgery) and the microscopic image 320b is obtained after the resection (or surgery). In addition, a radiological image or scan 334a from before the surgery and a radiological image or scan 334b ​​from after the surgery are shown. Such radiological images or scans can be used to improve the ground truth.

[0059] Both microscopic image 320a and microscopic image 320b are preferably captured with the same imaging settings, and microscopic image 320b is automatically changed if the settings are changed relative to the settings at which the image was acquired, for example, while tissue resection is being performed, to obtain an image with a similar field of view (FoV) to the image taken before the resection.

[0060] As previously mentioned, a tissue, for example the brain, may contain harmful tissue, for example a tumor, that must be removed during surgery. However, such tissue may also contain harmful tissue, for example a tumor, that should not be removed because it is located near or within a brain region responsible for some cognitive processes, such as speech. In this case, removing the tumor may damage the brain region and cause the patient to lose the ability to speak. Depending on the particular type or region in which a tumor or other harmful tissue is located, the decision of whether or not to remove it may also be influenced by patient data.

[0061] Such a determination of the action to be taken on the tissue is illustrated in Figure 3 with pre-resection microscopic image 320a and post-resection microscopic image 320b. Microscopic image 320a shows pre-resection tissue 370a (e.g., a brain or portion thereof) having therein portions 372, 374, 376, and 378. Portion 378 is part of the tumor core corresponding to portion 376.

[0062] In microscopic image 320b, which shows tissue 370b corresponding to tissue 370a but after resection, only portions 372 and 378, which were not resected during surgery, are visible. Portions 374 and 376 are no longer visible, as they were resected during surgery.

[0063] In the following, it is described how a surgeon or other qualified user or personnel can create annotations to be used as training data. Pairs of (pre- and post-resection) microscopic images 320a, 320b are loaded into the annotation application 330, for example from the database 122 shown in Fig. 1. In addition, (pre- and post-operative) radiological images or scans 334a, 334b ​​or radiological images obtained from radiological scans are also preferably loaded into the annotation application 330. In addition, ultrasound images, endoscopic images and / or neuromonitoring information can also be loaded into the annotation application 330.

[0064] The microscope images 320a, 320b and the radiological images (or scans) 334a, 334b ​​are mutually registered in order to receive images (scans) having the same field of view so that they can be compared with each other.

[0065] Furthermore, patient data of the patients for whom the microscopic images 320a, 320b are captured, such as age 336a (e.g. as averages) and blood pressure 336b, are provided to the annotation application 330. The patient data also includes, in particular, the life expectancy 336c of each patient after tissue resection has been performed, as well as parameters that "measure" the patient's quality of life after surgery, such as the rate of tumor recurrence and loss of some cognitive functions, e.g. in the form of categorical variables. The patient data also preferably includes measurements of tumor treatment (e.g. radiation therapy and / or chemotherapy doses).

[0066] The surgeon can then annotate the microscopic images based on what is visible before and after the resection, as well as his own knowledge and patient data. Note that such annotation can be done only based on the microscopic image 320a before the resection, since it shows all harmful tissue previously present, but instead the microscopic image 320b after the resection can also be used.

[0067] In the case shown in FIG. 3, portion 374 is not present in microscopic image 320b because it has been removed. The surgeon can determine that portion 374 is a tumor and has been correctly removed. The surgeon can create annotation 332a that includes information that portion 374 is a tumor and should, for example, be (positively) removed. Portion 376 is not present in microscopic image 320b and has been removed. The surgeon can determine that portion 376 is a tumor and has been correctly removed. The surgeon can create annotation 332b that includes information that portion 376 is a tumor and should, for example, be (positively) removed.

[0068] Portion 372 is also present in microscopic image 320b. The surgeon may determine that portion 372 was (or is) a tumor but was not removed for a particular reason. For example, this may be due to the age of the patient. The surgeon may, for example, create annotation 332c including information that portion 372 is a tumor but should not be removed. Similarly, annotation 332c here may include information that portion 372 is a tumor but should not be removed, for example, at a particular age of the patient.

[0069] Portion 378, a tumor core / portion 376, is also present in the microscopic image 320b. The surgeon can determine that portion 378 was (or is) a tumor but was not removed for a particular reason. For example, this may be due to the age of the patient. The surgeon can, for example, create annotation 332d including information that portion 378 is a tumor but should not be removed. Such annotation 332d can also include information that portion 378 is a tumor but should not be removed, for example, at a particular age of the patient.

[0070] Additional information, such as tissue categories or classes 380 shown in the microscopic images 320a, 320b, such as HGG (high grade glioma) nuclei, HGG dense rim, HGG low density rim, LGG (low grade glioma), or healthy anatomical structures 382, ​​such as arteries, veins, etc., can be displayed to assist the surgeon and provide annotation to assist in training machine learning algorithms.

[0071] For example, depending on the category or class 380 of the harmful tissue, e.g., portion 378, the surgeon can make annotations 332d that include information about whether portion 378 is a tumor and should be resected or not. Life expectancy 336c can also be taken into account, as described below.

[0072] Even with the complete definition of tumor tissue (core and margin, e.g., portions 378 and 376), it is still difficult to determine where to set the borders of resection to maximize life expectancy without compromising quality of life. Note that both life expectancy and quality of life after surgery depend on other health parameters (e.g., blood pressure, cholesterol level, etc.) and the above-mentioned diseases of the patient (patient data). All these parameters (tumor type and size, age, blood pressure, cholesterol level, various diseases, etc.) can be taken into account when trying to maximize what should be resected during surgery to maximize life expectancy without compromising quality of life.

[0073] In Fig. 4a, the instruction data generated by the application of a machine learning algorithm is shown in a schematic manner. On the left side of Fig. 4a, the instruction data 432a can be created or generated, for example, based on the annotations 332a, 332b, 332c, 332d created in the manner described with reference to Fig. 3, which shows the pre-resection microscopic image 320a as shown in Fig. 3, and further on the patient data 336a, 336b. On the right side of Fig. 4a, two different ways of how the instruction data 432a here can be created or presented are shown and will be described below. The instruction data 432a here can include boundaries, for example boundaries 472a, 478a.

[0074] The instruction data 432a indicates an action to be performed on the tissue, for example, the instruction data 432a may include information that a particular portion of the tissue should be excised, such as information that tissue within boundary 472a and boundary 478a should be excised. Such instruction data may be generated for all portions of tissue 370a.

[0075] As mentioned above, the instruction data 432a here can be formed or generated in the form of a boundary or resection boundary or some of such boundaries or to include a boundary or resection boundary or some of such boundaries. In one embodiment, such boundary 478a is determined, for example, from a number of boundaries 476a, 478a each having different values ​​of a parameter related to the tissue or the patient from which the tissue originates, so that the value of the parameter is maximized, especially when regression methods are used. This intermediate step is shown in the center of FIG. 4a. The boundary 478a corresponds, for example, to the portion 378, which is the center of the tumor. The boundary 476a corresponds, for example, to the portion 376, which is the margin or edge of a diffuse portion of the same tumor. Depending on the patient data, for example, both of the two boundaries 476a, 478a can be taken into account to define the tissue to be resected. However, when resecting the tissue within the boundary 476a (including the tissue within the boundary 478a), the patient may have a shorter life expectancy and / or a lower quality of life than if only the tissue within the boundary 478a was resected. Thus, for example, boundary 478a is selected to be included in prescription data 432a by maximizing reaming life span 336c using a machine learning algorithm trained to consider quality of life by having categorical variables such as tumor recurrence rate and loss of some cognitive function, etc. It should be noted that such boundary, indicating the area to be resected, may also encompass the area between lines 476a and 478a.

[0076] Fig. 4b is a schematic diagram of instruction data generated by application of a machine learning algorithm. On the left side of Fig. 4b, a pre-resection microscopic image 320a as shown in Fig. 3 is shown. Based on the annotations 332a, 332b, 332c, 332d created as described with reference to Fig. 3 and further on the patient data 336a, 336b, for example, instruction data 432b can be created or generated. Two different ways of how to create or present such instruction data 432b are shown on the right side of Fig. 4b and will be described below. Here, instruction data 432b can include an indicator, such as a color or a shade. In one embodiment, such an indicator is determined, for example, based on the characteristics of pixels in said image 320a.

[0077] Region 478b with indicator (shading) corresponds to portion 378 classified as a tumor core, for example, and should be excised. Similarly, region 472a with indicator (shading) has been classified as tumor and should be excised.

[0078] Such indication data 432a, 432b in the form of a boundary or in the form of an indicator (shaded area) can be presented to the surgeon during surgery to assist him in making a decision whether to resect a particular portion of tissue. It should be noted that the type of line indicating the boundary is merely an example. Other suitable formats for indicating the area to be resected can be selected. For example, the area within said boundary to be resected can be shaded, which can be particularly useful when the area to be resected is annular, etc.

[0079] As mentioned above, various ways are possible for how the instruction data 432a, 432b are presented or generated. In the top right corner of each of Figs. 4a and 4b, only the instruction data 432a, 432b in the form of two boundaries or areas 472, 478 are shown, which can be displayed, for example, next to the original microscope image 320a. In the bottom right corner of each of Figs. 4a and 4b, the instruction data 432a, 432b in the form of two boundaries 472a, 472a or indicators 472b, 478b are shown overlaid on the original microscope image 320a, which allows, for example, the surgeon to see which part of the tissue to remove.

[0080] In Fig. 5, a computer-implemented method for training a machine learning algorithm according to an embodiment of the present invention is illustrated in a flow chart, which shows in schematic form. In step 500, training data is received, which comprises images showing tissue. According to an embodiment of the present invention, the training data further comprises patient data, including information about a patient correlated with the tissue shown in the images, such as parameters such as age, sex, type and dose of tumor treatment, life expectancy, blood pressure, cholesterol level, etc. According to an embodiment of the present invention, the training data further comprises annotations for the images, as described with reference to Figs. 3 and 4a-b, which indicate at least one of the class or type of tissue shown in the images and the actions to be performed or performed on the tissue.

[0081] In step 502, a machine learning algorithm is tuned (trained) based on training data for tuning the machine learning algorithm such that (when subsequently applied) the machine learning algorithm generates instruction data 232 for at least a portion of the tissue shown in the image 120, where the instruction data 232 indicates an action to be taken on the tissue. Then, in step 504, the tuned (trained) machine learning algorithm is provided for application.

[0082] In Fig. 6, a computer-implemented method for providing instruction data relating to tissue shown in a microscopic image according to another embodiment of the present invention is illustrated by a flow chart. In step 600, input data is received. The input data comprises microscopic images, e.g. in real time, from a surgical microscope acquired during surgery. The microscopic images represent tissue of a patient. In step 602, instruction data 232, 432 is generated for at least a portion of the tissue shown in the microscopic image 220 by applying a machine learning algorithm, the instruction data being indicative of an action to be performed on the tissue. In step 604, output data comprising the instruction data is provided. In step 606, the instruction data can be displayed on a display.

[0083] In Fig. 7, a method for providing images and instruction data to a user according to another embodiment of the invention is illustrated generally by a flow chart. In step 700, tissue of a patient is illuminated. In step 702, a microscopic image of the tissue is captured by a surgical microscope (using an image sensor), where the microscopic image is indicative of the tissue. In step 704, instruction data is generated by applying a machine learning algorithm, for example as described with respect to Figs. 2-6. In step 706, the microscopic image and instruction data are provided to a user of the surgical microscope, for example on a display.

[0084] As used in this specification, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0085] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0086] Some embodiments relate to a microscope including a system as described in relation to one or more of the figures of Figures 1 to 7. Alternatively, the microscope may be part of a system as described in relation to one or more of the figures of Figures 1 to 7 or may be connected to a system as described in relation to one or more of the figures of Figures 1 to 7. Figure 2 shows a schematic diagram of a system configured to perform the methods described herein. The system includes a microscope 200 and a computer system 250. The microscope 200 is configured to take images and is connected to the computer system 250. The computer system 250 is configured to perform at least some of the methods described herein. The computer system 250 may be configured to execute machine learning algorithms. The computer system 250 and the microscope 200 may be separate entities, but may be integrated in one common housing. The computer system 250 may be part of a central processing system of the microscope 200 and / or the computer system 250 may be part of a subordinate part of the microscope 200, such as a sensor, actor, camera or lighting unit of the microscope 200.

[0087] The computer system 150 or 250 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, such as local clients and / or one or more remote server farms and / or data centers). The computer system 150 or 250 may include any circuit or combination of circuits. In one embodiment, the computer system 150 or 250 may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit, for example, of a microscope or a microscope component (e.g., a camera). Other types of circuits that may be included in computer system 150 or 250 may be custom circuits, application specific integrated circuits (ASICs), etc., such as one or more circuits (such as communications circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems.Computer system 720 may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives handling removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc. Computer system 250 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touch screen, voice recognition device, or any other device that enables a user of the system to input information to and receive information from computer system 150 or 250.

[0088] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, microprocessor, programmable computer, or electronic circuitry. In some embodiments, any one or more of the essential steps may be performed by such an apparatus.

[0089] Depending on certain implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementation can be performed by a non-transitory recording medium, such as a digital recording medium, for example a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM and EPROM, an EEPROM or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Thus, the digital recording medium can be computer readable.

[0090] Some embodiments of the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to perform any of the methods described herein.

[0091] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code which is operable to perform any of the methods when the computer program product is run on a computer, the program code may for example be stored on a machine readable carrier.

[0092] Another embodiment comprises the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0093] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein, when the computer program runs on a computer.

[0094] Therefore, another embodiment of the present invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein, including a processor and a recording medium.

[0095] A further embodiment of the invention is therefore also a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or the sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0096] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0097] Another embodiment comprises a computer having the computer program installed thereon for performing any of the methods described herein.

[0098] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server to transfer the computer program to the receiver.

[0099] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus.

[0100] The embodiments may be based on the use of machine learning models or algorithms. Instead of relying on models and inferences, machine learning may refer to algorithms and statistical models that a computer system may use to perform a particular task without using explicit instructions. For example, machine learning may use data transformations inferred from analysis of past data and / or training data instead of rule-based data transformations. For example, image content may be analyzed using a machine learning model or using a machine learning algorithm. For a machine learning model to analyze image content, the machine learning model may be trained with training images as input and training content information as output. By training the machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize image content, such that image content not included in the training data becomes recognizable using the machine learning model. The same principle may be used for other types of sensor data in a similar manner: by training the machine learning model with training sensor data and a desired output, the machine learning model "learns" a transformation between sensor data and output, which can be used to provide an output based on the non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata and / or image data) may be pre-processed to obtain feature vectors that are used as input to a machine learning model.

[0101] The machine learning model may be trained using training input data. The above example uses a training method called "supervised learning". In supervised learning, the machine learning model is trained using multiple training samples, where each sample may include multiple input data values ​​and multiple desired output values, i.e., each training sample is associated with a desired output value. By specifying both the training samples and the desired output value, the machine learning model "learns" during training which output value to provide based on input samples that are similar to the provided sample. Besides supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on supervised learning algorithms (e.g. classification algorithms, regression algorithms or similarity learning algorithms). Classification algorithms may be used when the output is restricted to a limited set of values ​​(categorical variables), i.e., the input is classified into one of a limited set of values. Regression algorithms may be used when the output may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms, but are based on learning from examples with a similarity function that measures how similar or related two objects are. In addition to supervised or semi-supervised learning, unsupervised learning may be used to train machine learning models. In unsupervised learning, input data may be (only) provided, and unsupervised learning algorithms may be used to find structure in the input data (e.g., by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data containing multiple input values ​​into multiple subsets (clusters), such that input values ​​in the same cluster are similar according to one or more (predefined) similarity criteria, but are not similar to input values ​​contained in another cluster.

[0102] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning may be used to train machine learning models. In reinforcement learning, one or more software actors (referred to as "software agents") are trained to take actions in their surroundings. Based on the actions taken, rewards are calculated. Reinforcement learning is based on training one or more software agents to select actions that result in an increasing cumulative reward (as manifested by an increasing reward), resulting in the software agent becoming better at a given task.

[0103] Furthermore, some techniques may be applied to parts of the machine learning algorithm. For example, feature representation learning may be used. In other words, the machine learning model may be trained at least in part with feature representation learning and / or the machine learning algorithm may include a feature representation learning component. A feature representation learning algorithm, which may be referred to as a representation learning algorithm, may not only preserve information in its input, but may also transform the information to make it useful, often as a pre-processing step before performing classification or prediction. Feature representation learning may be based on, for example, principal component analysis or cluster analysis.

[0104] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide identification of input values ​​that raise suspicion by differing significantly from the majority of the input or training data. In other words, a machine learning model may be trained at least in part with anomaly detection and / or a machine learning algorithm may include an anomaly detection component.

[0105] In some examples, the machine learning algorithm may use a decision tree as a predictive model. In other words, the machine learning model may be based on a decision tree. In a decision tree, an observation about an item (e.g., a set of input values) may be represented by a branch of the decision tree, and an output value corresponding to this item may be represented by a leaf of the decision tree. The decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree may be represented as a classification tree, and if continuous values ​​are used, the decision tree may be represented as a regression tree.

[0106] Association rules are another technique that may be used in machine learning algorithms. In other words, a machine learning model may be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. A machine learning algorithm may identify and / or utilize one or more association rules that represent knowledge derived from the data. These rules may be used, for example, to store, manipulate, or apply the knowledge.

[0107] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions that may be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a set of data structures and / or rules that represent learned knowledge (e.g., based on training performed by a machine learning algorithm). In embodiments, the use of machine learning algorithm may refer to the use of an underlying machine learning model (or underlying machine learning models). The use of machine learning model may refer to the machine learning model and / or the set of data structures / rules that are the machine learning model being trained by a machine learning algorithm.

[0108] For example, the machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or the brain. An ANN contains a number of interconnected nodes and a number of junctions, so-called edges, between the nodes. Typically, there are three types of nodes: input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may convey information from one node to another. The output of a node may be defined as a (non-linear) function of its inputs (e.g. the sum of its inputs). The inputs of a node may be used in a function based on the "weights" of the edges or nodes that provide the inputs. The weights of the nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input.

[0109] Alternatively, the machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that may be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing input with multiple training input values ​​that belong to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, the machine learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may represent a set of random variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machine learning model may be based on a genetic algorithm, which is a heuristic method that mimics search algorithms and the process of natural selection. [Explanation of symbols]

[0110] 100,200 Surgical microscope 102,104,202,204 Illumination optical system 106,206 Image Sensor 110,210 Surgeon 112,212 patients 118,218 Raw Data 120,220,320a,320b Microscope images 122,140 databases 130,330 Annotation Applications 132,332a,332b,332c,332d Annotation 134,334a,334b Radiological images 136,336a,336b Patient Data 142 Training Data 150,250 systems 152,252 processors 154,254 storage devices 160 Machine Learning Algorithms 164,264 trained machine learning algorithms 224 Corrected Microscope Images 232,432a,432b Instruction data 270 Display 370a,370b Organization 372,374,376,378 Organization part 380,382 Information 472a,476a,478a boundary 472b,478b indicator 500~504,600~606,700~706 Method steps

Claims

1. 1. A system (150) for training a machine learning algorithm (160), comprising one or more processors (152) and one or more storage devices (154), the system (150) comprising: receiving training data (142) including images (120, 320a, 320b) showing tissue; adjusting the machine learning algorithm (160) based on the training data (142) such that the machine learning algorithm generates prescription data (232, 432a, 432b) for at least a portion of the tissue shown in the image (120, 320a, 320b) indicating an action to be performed on the tissue during surgery, the prescription data (232, 432a) including a resection boundary (472a, 478a) indicating an area of ​​the tissue within which the action on the tissue is to be performed; providing a trained machine learning algorithm (164); It is configured as follows: System (150).

2. the training data (142) further includes patient data (136) containing information about a patient correlating to the tissue shown in the image (120); The system (150) of claim 1.

3. The patient data (136) includes at least one of the following parameters for the patient: age, sex, type and dose of tumor treatment, life expectancy, blood pressure, cholesterol level, tumor recurrence rate, loss of at least part of one or more cognitive functions; The system (150) of claim 2.

4. the machine learning algorithm (160) is adapted to generate the prescription data (232, 432a, 432b) such that the prescription data is based at least in part on at least a portion of the patient data (136) to be provided as input data to the trained machine learning algorithm (164) for application; The system (150) of claim 2 or 3.

5. the training data (142) further includes annotations (132) for the images; the annotation indicates at least one of a class or type of tissue shown in the image (120) and an action to be performed or performed on the tissue; The system (150) of any one of claims 1 to 3.

6. The trained machine learning algorithm (164) is obtained based on supervised learning. The system (150) of any one of claims 1 to 3.

7. the machine learning algorithm (164) bases the classification on at least a portion of the annotations for the image; The system (150) of claim 5.

8. the machine learning algorithm (164) is regression-based; The system (150) of claim 6.

9. The trained machine learning algorithm is obtained based on unsupervised learning. The system (150) of any one of claims 1 to 3.

10. the images (120, 320a, 320b) showing the tissue comprise microscopic images from a surgical microscope (100) acquired during surgery; The system (150) of any one of claims 1 to 3.

11. the training data (142) further includes at least one of radiological images (134) or scans, ultrasound images, endoscopic images, and neuromonitoring information; The system (150) of any one of claims 1 to 3.

12. the images showing the tissue further include at least one of a radiological image (134), an ultrasound image, and an endoscopic image, each showing tissue corresponding to the tissue shown in the microscopic image and having a field of view that is the same as or similar to the field of view of the corresponding microscopic image; The system (150) of claim 10.

13. the system (150) is configured to determine, from a plurality of boundaries (476a, 478a) each having a different value of a parameter related to the tissue or related to a patient from which the tissue originates, the boundary (478a) such that the value of the parameter is maximized. The system (150) of any one of claims 1 to 3.

14. the instruction data (232, 432b) includes different indicators (472b, 478b) for different actions to be taken on the tissue; The system (150) of any one of claims 1 to 3.

15. the machine learning algorithm (160) is adapted to generate the indication data (232, 432a, 432b) overlaid on the microscopic image (120, 220, 320a); The system (150) of any one of claims 1 to 3.

16. 1. A computer-implemented method for training a machine learning algorithm (160), the method comprising: receiving (500) training data (142) including images (120, 320a, 320b) showing tissue; adjusting (502) the machine learning algorithm (160) based on the training data (142) such that the machine learning algorithm generates prescription data (232, 432) for at least a portion of the tissue shown in the image (120, 320a, 320b) indicating an action to be performed on the tissue during surgery, the prescription data (232, 432a) including a resection boundary (472a, 478a) indicating a region of the tissue within which the action is to be performed on the tissue; Providing (504) a trained machine learning algorithm (164); A method comprising:

17. A trained machine learning algorithm (164), receiving training data (142) including images (120, 320a, 320b) showing tissue; adjusting the machine learning algorithm (160) based on the training data (142) such that the machine learning algorithm generates prescription data (232, 432) for at least a portion of the tissue shown in the image (120) that indicates an action to be performed on the tissue during surgery, the prescription data (232, 432a) including a resection boundary (472a, 478a) that indicates a region of the tissue within which the action is to be performed on the tissue; Trained by A trained machine learning algorithm (164).

18. 1. A system (250) for providing indicative data (232, 432a, 432b) relating to tissue shown in a microscopic image (220), the system (250) comprising one or more processors (252) and one or more storage devices (254), the system (250) comprising: receiving input data including microscopic images (220) from a surgical microscope (200) showing tissue of a patient, the images being acquired during surgery; applying a machine learning algorithm (264) to generate prescription data (232, 432a, 432b) for at least a portion of the tissue shown in the microscopic image (220) indicating an action to be performed on the tissue during surgery, the prescription data (232, 432a) including a resection boundary (472a, 478a) indicating an area of ​​the tissue within which the action on the tissue is to be performed; providing output data including indication data (232) for said microscopic image (220); It is configured as follows: System (250).

19. the input data further includes patient data (236), and the prescription data (236) is determined at least in part based on at least a portion of the patient data (236). The system (250) of claim 18.

20. the output data includes the microscope image (220) overlaid with the instruction data (232, 432a, 432b); 20. The system (250) of claim 18 or 19.

21. 20. The system (250) of claim 18 or 19, wherein a trained machine learning algorithm (164) of claim 17 is used.

22. A surgical microscope system comprising a surgical microscope (200), an image sensor (206) and a system (250) according to claim 18 or 19.

23. 1. A computer-implemented method for providing indicative data (232) related to tissue shown in a microscopic image (220), the method comprising: receiving (600) input data including a microscopic image (220) showing tissue of a patient from a surgical microscope (200) acquired during surgery; applying a machine learning algorithm (264) to generate (602) prescription data (232, 432a, 432b) for at least a portion of the tissue shown in the microscopic image (220) that indicates an action to be performed on the tissue during surgery, the prescription data (232, 432a) including a resection boundary (472a, 478a) that indicates an area of ​​the tissue within which the action is to be performed on the tissue; providing (604) output data including the indication data (232, 432a, 432b) for the microscopic image (220); A method comprising:

24. 1. A method for providing an image (220) and instructional data to a user (210) using a surgical microscope (200), the method comprising: illuminating (700) tissue of a patient (212); capturing (702) a microscopic image (220) of the tissue showing the tissue; applying a machine learning algorithm (164, 264) to generate (704) prescription data (232, 432a, 432b) for at least a portion of the tissue shown in the microscopic image, the prescription data (232, 432a) including a resection boundary (472a, 478a) indicating an area of ​​the tissue within which the action is to be performed on the tissue; providing (706) the microscopic image and the instruction data to a user (210) of the surgical microscope (200); A method comprising:

25. A computer program comprising a program code for performing the method according to claim 16 or 23 when the computer program is run on a processor.

Images