System, microscope system, method, and computer program for training or using a machine learning model

Abstract

The embodiments relate to a system, method, and computer program for training a machine learning model, as well as to a machine learning model, method, and computer program for detecting at least one property of an organic tissue sample, and further to a microscope system. The system includes one or more storage modules and one or more processors. The system is configured to acquire multiple images of a sample of organic tissue. The multiple images are taken using multiple different imaging properties. The system is configured to train a machine learning model using the multiple images. The multiple images are used as training samples, and information regarding the at least one property of the organic tissue sample is used as a desired output of the machine learning model. The machine learning model is trained such that the machine learning model is suitable for detecting the at least one property of the organic tissue sample in image input data that represents (only) a proper subset of the multiple different imaging properties. The system is configured to provide the machine learning model.

Description

Technical Field

[0001] Embodiments relate to systems, methods, and computer programs for training a machine learning model, and also to machine learning models, methods, and computer programs for detecting at least one characteristic of a sample of an organic tissue, and further to a microscope system.

Background Art

[0002] The main use of a microscope is in the analysis of organic tissues. For example, a microscope can be used to obtain a detailed view of an organic tissue, allowing a general practitioner and a surgeon to detect characteristics of the tissue, such as abnormal (i.e., "pathological") tissues, among normal organic tissues.

Summary of the Invention

Problems to be Solved by the Invention

[0003] There may be a need for an improved approach for the analysis of organic tissues that enables better detection of tissue characteristics.

Means for Solving the Problems

[0004] This need is addressed by the subject matter of the independent claims.

[0005] Embodiments of the present disclosure provide a system that includes one or more storage modules and one or more processors. The system is configured to acquire a plurality of images of a sample of an organic tissue. The plurality of images are captured using a plurality of various imaging characteristics. The system is configured to train a machine learning model using the plurality of images. The plurality of images are used as training samples, and information regarding at least one characteristic of the sample of the organic tissue is used as a desired output of the machine learning model. The machine learning model is trained such that the machine learning model is suitable for detecting at least one characteristic of the sample of the organic tissue in image input data that reproduces only an appropriate subset of the plurality of various imaging characteristics. The system is configured to supply the machine learning model.

[0006] Predetermined features of organic tissues, such as the shape of features or abnormal tissues, can be more easily detected in images taken with various image characteristics. For example, in a predetermined spectral band, the reflection, fluorescence, or bioluminescence of a part of the organic tissue can be made characteristic with respect to abnormal tissues. By using a plurality of images of the same organic tissue taken using various image characteristics (e.g., in various spectral bands, in various imaging modes, using various polarizations, etc.), it is possible to train a machine learning model that can provide an artificial intelligence form to estimate the expression of at least one feature such as an abnormal tissue even from an image that matches only an appropriate subset of various image characteristics. For example, as image characteristics, in addition to a white light reflection image (color image) having a visible light spectrum or reflection imaging, additional images can be used as training samples taken in a spectral band where the characteristics of abnormal tissues or the like stand out due to their reflection or fluorescence. The machine learning model can "learn" to detect characteristics using input samples taken using a plurality of imaging characteristics, whereby even if input data that reproduces only a subset of the imaging characteristics is supplied to the machine learning model, it is still possible to detect features such as abnormal tissues or normal tissues.

[0007] According to an embodiment of the present disclosure, further provided is a method for training a machine learning model. The method includes obtaining a plurality of images of a sample of an organic tissue. The plurality of images are taken using a plurality of various imaging characteristics. The method includes training a machine learning model using the plurality of images. The plurality of images are used as training samples, and information regarding at least one characteristic of the sample of the organic tissue is used as a desired output of the machine learning model in the training of the machine learning model. The machine learning is trained such that the machine learning model is suitable for detecting at least one characteristic of the sample of the organic tissue in image input data that reproduces an appropriate subset of the plurality of various imaging characteristics. The method includes providing the machine learning model. According to an embodiment of the present disclosure, further provided is a machine learning model trained using a system or method.

[0008] According to an embodiment of the present disclosure, further provided is a method for detecting at least one characteristic of a sample of an organic tissue. The method includes using the machine learning model generated by the above-described system or method together with image input data that reproduces an appropriate subset of the plurality of various imaging characteristics.

[0009] According to an embodiment, further provided is a computer program including program code for executing at least one of the above-described methods when the computer program is executed in a processor.

[0010] Such an embodiment can be used, for example, in a microscope such as a surgical microscope to assist in detecting at least one feature during surgery. According to an embodiment of the present disclosure, provided is a microscope system including the above-described system or configured to execute at least one of the plurality of methods.

[0011] Hereinafter, some embodiments of the apparatus and / or method will be described by way of example only with reference to the accompanying drawings.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2a

Figure 2b

Figure 3

Figure 4a

Figure 4b

Figure 5a

Figure 5b

Figure 5c

Figure 6a

Figure 6b

Modes for Carrying Out the Invention

[0013] Next, various embodiments will be described in more detail with reference to the accompanying drawings showing some examples. For clarity, in the figures, the thickness of lines, layers, and / or regions may sometimes be emphasized.

[0014] Thus, further embodiments are capable of various modifications and alternative forms, some of which specific embodiments are shown in the drawings and will be described in detail below. However, this detailed description is not intended to limit further embodiments to the specific forms described. Further embodiments can cover any modifications, equivalents, and alternatives falling within the scope of the present disclosure. The same or similar reference numerals refer to the same or similar elements throughout the description of the drawings, and those elements can be implemented in the same or in a modified form while providing the same or similar functions and being contrasted with each other.

[0015] Obviously, if an element is referred to as being "connected" or "coupled" to another element, those elements may be directly connected or coupled, or may be connected or coupled through one or more intervening elements. If two elements A and B are coupled using "or", this is to be understood as disclosing all possible combinations, i.e., A only, B only, as well as A and B, unless explicitly or implicitly defined otherwise. Alternative phrasings for the same combination are "at least one of A and B" or "A and / or B". The same applies to combinations of more than two elements.

[0016] The terms used herein for the purpose of describing particular embodiments are not intended to limit further embodiments. Whenever singular forms such as definite articles, indefinite articles, etc. are used and it is not explicitly or implicitly defined as essential to use only a single element, additional embodiments can also use multiple elements to achieve the same function. Similarly, if a function is subsequently described as being achieved using multiple elements, additional embodiments can use a single element or processing entity to achieve the same function. Furthermore, as is obvious, the terms "having", "having", "including", and / or "including" when used, specify the presence of the described features, wholes, steps, operations, processes, behaviors, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, processes, behaviors, elements, components, and / or any group thereof.

[0017] Unless defined otherwise, all terms (including technical and scientific terms) are used herein in their ordinary meaning in the field to which the embodiments belong.

[0018] FIG. 1 shows a block diagram of one embodiment of a system 100 for training a machine learning model. The system includes one or more storage modules 110 and one or more processors 120 coupled to the one or more storage modules 110. Optionally, the system 100 includes one or more interfaces 130 that can be coupled to the one or more processors 120 for obtaining and / or supplying information, for example, for supplying a machine learning model and / or for obtaining multiple images. Generally, one or more processors 120 of the system can be configured to perform the following tasks in cooperation with, for example, the one or more storage modules 110 and / or the one or more interfaces 130.

[0019] The system is configured to acquire multiple images of one sample of an organic tissue. The multiple images are taken using multiple different imaging characteristics. The system is configured to train a machine learning model using the multiple images. The multiple images are used as training samples. Information regarding at least one characteristic of the sample of the organic tissue is used as a desired output of the machine learning model. The machine learning model is trained to be suitable for detecting at least one characteristic of the sample of the organic tissue in image input data that reproduces (only) a suitable subset (and not the whole, for example, of the multiple different imaging characteristics) of the multiple different imaging characteristics. The system is configured to supply the machine learning model. For example, the system can be a system implemented by a computer.

[0020] According to an embodiment, a system, a method, and a computer program for training a machine learning model are provided. Machine learning can be said to refer to algorithms and statistical models that can be used by a computer system to perform a specific task by relying on models and inferences instead of using explicit instructions. For example, in machine learning, instead of rule-based data transformation, data transformation inferred from the analysis of past data and / or training data can be used. For example, image content can be analyzed using a machine learning model or using a machine learning algorithm. In order for a machine learning model to analyze image content, the machine learning model can be trained using training images as input and training content information as output. By training the machine learning model using a large number of training images and / or training sequences (for example, words or sentences) and related training content information (for example, labels or annotations), the machine learning model "learns" to recognize image content, and thus image content not included in the training data can be recognized using the machine learning model.

[0021] A machine learning model can be trained using training input data. The above example uses a training method called "supervised learning". In the case of supervised learning, the machine learning model is trained using a plurality of training samples, where each sample can include a plurality of input data values and a plurality of desired output values, that is, each training sample is associated with a desired output value. By specifying both the training sample and the desired output value, the machine learning model "learns" which output value to share based on input samples similar to the samples supplied during training.

[0022] According to an embodiment, this approach can be used in multiple images. In other words, the machine learning model can be trained using supervised learning. Multiple images are supplied to the machine learning model as training samples. For example, multiple images can be input, for example, simultaneously, at multiple inputs of the machine learning model. As the corresponding desired output, information regarding at least one characteristic can be used. For example, information regarding at least one characteristic of a sample of an organic tissue can represent at least one part of a sample of a normal or abnormal organic tissue. Thus, information regarding an abnormal or normal tissue can be used as the desired output for training the machine learning model. In this case, the machine learning model can be trained so as to be suitable for detecting an abnormal or normal tissue in image input data that reproduces a suitable subset of multiple various imaging characteristics. Alternatively or additionally, information regarding at least one characteristic of a sample of an organic tissue can represent the shape of one or more features (such as blood vessels, separate parts of the sample of the organic tissue, bone structure, etc.) of the sample of the organic tissue. Thus, information regarding the shape of one or more features of a sample of an organic tissue can be used as the desired output for training the machine learning model. In this case, the machine learning model can be trained so as to be suitable for detecting the shape of one or more features in image input data that reproduces a suitable subset of multiple various imaging characteristics. Generally, information regarding at least one characteristic of a sample of an organic tissue, such as information regarding an abnormal or normal tissue, or information regarding the shape of one or more features of a sample of an organic tissue, can be made to correspond to an image or bitmap in which the part of the sample of the organic tissue representing at least one characteristic of the sample of the organic tissue is highlighted or indicated. To improve the results of machine learning, the image or bitmap can have the same size as the image among the multiple images, or can have at least the same aspect ratio, and / or can represent the same segment of the sample of the organic tissue.To improve the training accuracy of a machine learning model and thus improve the values obtained by using captured images having a plurality of various image characteristics, a) a plurality of images are accurately aligned with each other such that a part of an organic tissue shown in a pixel in a first image is also shown in a corresponding pixel of a second image among the plurality of images, and b) the images can be captured substantially simultaneously, thereby ensuring that, for example, the organic tissue does not change between the images. In other words, the system can be configured to correlate (i.e., accurately align) the plurality of images on a pixel-by-pixel basis. The machine learning model can be trained based on the correlated plurality of images. Additionally or alternatively, the plurality of images can be images recorded substantially simultaneously. In other words, the plurality of images can be captured within a maximum of 30 seconds (or a maximum of 15 seconds, a maximum of 10 seconds, a maximum of 5 seconds, a maximum of 2 seconds, a maximum of 1 second) with respect to each other (applied to each image pair of the plurality of images described above).

[0023] The system is configured to acquire a plurality of images of one sample of an organic tissue. In biology, a tissue is an aggregate of similar cells (and extracellular matrix) that have the same origin and perform a specific function together. The term "organic" tissue means that the tissue is part of or derived from an organism such as an animal, a human, or a plant. For example, the organic tissue can be (human) brain tissue, and the machine learning model can be trained to detect a brain tumor (abnormal tissue that is a brain tumor). For example, for the same sample of an organic tissue, a plurality of images can be captured from the same angle. The plurality of images can show the same segment of the sample of the organic tissue. The plurality of images can be accurately aligned with each other such that a part of the organic tissue shown in a pixel in a first image (after correlation of the plurality of images, for example) is also shown in a corresponding pixel of a second image among the plurality of images. According to at least some embodiments, the plurality of images are a plurality of microscopic images, i.e., a plurality of images captured by a microscope camera.

[0024] The plurality of images are captured using a plurality of various imaging characteristics. In this context, the term "imaging characteristics" means that the plurality of images are captured using various techniques, and as a result, images having various characteristics can be obtained even for the same organic tissue (substantially simultaneously). For example, the plurality of images can be captured in various spectral bands, using various imaging modes (where these imaging modes are at least two of reflection imaging, fluorescence imaging, and bioluminescence imaging), using various polarizations (such as circular polarization, linear polarization, linear polarization at various angles), and further as various images at various time points in a time-resolved imaging sequence. In other words, the plurality of imaging characteristics can be related to at least one of various spectral bands, various imaging modes, various polarizations, and various time points in a time-resolved imaging sequence. Thus, the plurality of images can include one or more elements of a group consisting of microscopic images captured in various spectral bands, microscopic images captured in various imaging modes, microscopic images captured in various polarizations, and microscopic images representing various time points in a time-resolved imaging sequence.

[0025] Spectral images captured in various spectral bands can be made into images where the wavelength ranges of the light reproduced by the images (i.e., the "bands") are different respectively. This can be achieved by using various sensors (for example, using sensors that are sensitive only to a predetermined wavelength range), by placing various filters in front of the sensors (where the various filters filter various wavelength ranges), or by irradiating a sample of the organic tissue with light of various wavelength ranges.

[0026] By using various spectral bands, various imaging modes can be realized, such as reflectance imaging, fluorescence imaging or bioluminescence imaging. In the case of reflectance imaging, light is reflected by a sample of organic tissue at the same wavelength used to irradiate the sample of organic tissue, and the reflected light is reproduced by individual images. In the case of fluorescence imaging, light is emitted by a sample of organic tissue at a wavelength (or wavelength range) different from the wavelength (or wavelength range) used to irradiate the sample of organic tissue, and the emitted light is reproduced by individual images. In the case of bioluminescence imaging, the sample of tissue is not irradiated, yet it emits light, and this light is reproduced by individual images. In reflectance imaging, fluorescence imaging or bioluminescence imaging, one or more filters can be used to limit the wavelength range reproduced by individual images. Hereinafter, most of the examples relate to the use of various spectral bands and / or various imaging modes.

[0027] According to various embodiments, various spectral bands are used to detect fluorescent dyes that represent abnormal tissue or are applied to samples of organic tissue. For example, as external fluorescent dyes, fluorescein, indocyanine green (ICG), or 5-ALA (5-aminolevulinic acid) can be used. In other words, at least one subset of the images can be based on either the use of an external fluorescent dye or the autofluorescence of a sample of organic tissue. The fluorescent dye can be applied to a portion of the sample of organic tissue that is abnormal tissue, and thus, for example, the fluorescent dye can be distinguished in at least one of a plurality of images taken in the corresponding spectral band. In addition, in some cases, normal tissue or abnormal tissue, or a predetermined characteristic of a sample of organic tissue, can be made autofluorescent, and thus that characteristic can also be distinguished in at least one of a plurality of images taken in the corresponding spectral band. Thus, at least one subset of the plurality of images can reproduce a spectral band adjusted to at least one external fluorescent dye applied to the sample of organic tissue, for example, the spectral band in which light is emitted by a portion of the organic tissue to which the fluorescent dye is applied. Additionally or alternatively, at least one subset of the plurality of images can reproduce a spectral band adjusted to the autofluorescence of at least a portion of the sample of organic tissue, for example, the spectral band in which light is emitted by a portion of the organic tissue having autofluorescent properties.

[0028] Generally, a plurality of images can include one or more reflection spectral images and one or more fluorescence spectral images. For example, one or more reflection spectral images can reproduce the visible light spectrum. Each of the one or more fluorescence spectral images can reproduce a spectral band adjusted to the fluorescence at a specific wavelength observable in a sample of organic tissue. As a result, the plurality of images can include a subset of images (i.e., one or more fluorescence spectral images) that can better distinguish at least one characteristic of the sample of organic tissue, and a further subset of images that are likely to be used as input data in detecting at least one characteristic. For example, the plurality of images can include a subset of images (i.e., one or more fluorescence spectral images) that can better distinguish abnormal tissue from normal tissue, and a further subset of images that are likely to be used as input data in detecting normal or abnormal tissue. Additionally or alternatively, the plurality of images can include a subset of images (i.e., one or more fluorescence spectral images) that can better distinguish the shape of one or more features, and a further subset of images that are likely to be used as input data in detecting the shape of one or more features.

[0029] Additionally or alternatively, various polarizations can be used. Using various polarizations, the direction or angle in which the light incident on the camera is reproduced by individual images can be restricted. When using various polarizations, the plurality of images can include one or more elements among one or more images taken without polarization, one or more images taken using circular polarization, one or more images taken using linear polarization, and various images taken at various angles of linear polarization.

[0030] According to some embodiments, various time-resolved images can be used. For example, various images at various time points in a time-resolved imaging series can be used for a plurality of images. In the case of a time-resolved imaging series, for example, after irradiating a sample of organic tissue with light of a predetermined wavelength / band, the luminescence or fluorescence of the sample of organic tissue is recorded over a period (e.g., 1 second) until a certain luminescence effect or fluorescence effect appears.

[0031] As pointed out above, the plurality of images include images taken using a plurality (e.g., at least two, at least three, at least five, at least eight, at least ten) of various imaging characteristics. So far, the images have been two-dimensional images. In other words, the plurality of images can be two-dimensional images.

[0032] In some cases, since some abnormal tissues may be detectable due to their characteristic three-dimensional shape or surface structure, it may also be beneficial to include three-dimensional data as well. Thus, the plurality of images can include one or more three-dimensional representations of the sample of organic tissue. One or more three-dimensional representations of the sample of organic tissue can include a three-dimensional surface representation of the sample of organic tissue and / or a (microscopic) imaging tomography-based three-dimensional representation of the sample of organic tissue. For example, one or more three-dimensional representations of the sample of organic tissue can be accurately aligned with the two-dimensional images of the plurality of images. Additionally or alternatively, one or more three-dimensional representations of the sample of organic tissue can show the same segment of the sample of organic tissue as two-dimensional images of the plurality of images.

[0033] According to various embodiments, one of a plurality of images can be used as a desired output of a machine learning model or to determine a desired output, i.e., to determine information about at least one characteristic of a sample of an organic tissue (e.g., information about abnormal or normal tissue, or information about the shape of one or more features). Thereby, it is possible to train a machine learning model without the need for human annotation for one or more images of a sample of an organic tissue, or without the need to manually define at least one characteristic of a sample of an organic tissue. In other words, the information about at least one characteristic of a sample of an organic tissue can be based on one of the following one "reference image" (or a plurality of "reference images") of a plurality of images. Additionally or alternatively, the information about at least one characteristic of a sample of an organic tissue can be based on a three-dimensional representation of the sample of the organic tissue. The reference image can be processed to obtain (i.e., determine or generate) information about at least one characteristic of a sample of an organic tissue. In other words, the system can be configured to process an image to obtain information about at least one characteristic of a sample of an organic tissue (e.g., information about abnormal or normal tissue or information about the shape of one or more features).

[0034] When determining which of a plurality of images to select as a reference image, care must be taken. Generally, it is possible to select an image in which at least one characteristic is clearly distinguishable or visible. In other words, imaging characteristics representing specific characteristics of a sample of an organic tissue can be used to capture an image, for example, representing a specific type of abnormal tissue (or normal tissue), or representing the shape of one or more features of a sample of an organic tissue. As described above, fluorescence imaging may be used to obtain such an image. In other words, the reference image can be an image captured using fluorescence spectral imaging, that is, fluorescence imaging. In such a case, this image can be excluded as a training sample, for example, to avoid skewing of the machine learning model such that it only operates on input data captured using the same imaging characteristics.

[0035] In some cases, multiple characteristics may be detected. For example, there may be multiple types of abnormal tissues or multiple types of features. In this case, for example, multiple reference images among the plurality of images can be used and / or processed to obtain information regarding at least one characteristic of a sample of an organic tissue. In other words, the information regarding at least one characteristic of a sample of an organic tissue can be based on two or more (reference) images among the plurality of images. Imaging characteristics representing specific characteristics of a sample of an organic tissue can be used to capture each of two or more images, for example, representing a specific type of abnormal tissue (or normal tissue), or representing the shape of one or more features of a sample of an organic tissue. As a result, the information regarding at least one characteristic can be based on multiple types of characteristics, for example, based on multiple types of abnormal tissues, or based on multiple types of features. In other words, in the information regarding at least one characteristic, multiple different types of characteristics of a sample of an organic tissue can be highlighted or indicated, for example, separately or in combination.

[0036] The machine learning model is trained such that the machine learning model is suitable for detecting at least one characteristic of a sample of an organic tissue in image input data that reproduces a suitable subset of a plurality of various imaging characteristics. In other words, the input data can cover (i.e., reproduce) fewer imaging characteristics than the plurality of images used as training samples for the machine learning model. For example, the image input data can be the image input data of a camera operating within the visible light spectrum, or the image input data of a camera operating within the visible light spectrum, and the image input data can further include one, two, or three additional reflected images, fluorescence images, or bioluminescence images. For example, the camera can be a camera of a microscope, such as the camera of the microscope 310 in FIG. 3. When only a subset (only) of a plurality of characteristics, such as only the image input data of a camera operating within the visible light spectrum, is supplied to the machine learning model as input, the machine learning model can be trained such that the detection of at least one characteristic results in a (reliable) result. In some cases, the machine learning model can be used in situations where a fluorescent dye cannot be used for safety or cost reasons. Therefore, the image input data can be obtained from a tissue that has not been treated with an external fluorescent dye.

[0037] According to various embodiments, a single sample of an organic tissue may not be sufficient to properly train the machine learning model. For this reason, a plurality of samples of the organic tissue can be used together with a plurality of sets consisting of a plurality of images. For example, each set consisting of a plurality of images among the plurality of sets can be used together with corresponding information regarding at least one characteristic of a sample of the organic tissue. For example, the machine learning model can be trained by a single set of a plurality of images applied to the input of the machine learning model and corresponding information regarding at least one characteristic used as a desired output of the machine learning model.

[0038] As described above, at least one characteristic of a sample of an organic tissue can be detected using a machine learning model, for example, a normal tissue or an abnormal tissue in image input data or the shape of one or more features can be detected. In other words, a machine learning model is used together with image input data that reproduces a (suitable) subset of a plurality of different imaging characteristics, and the system can be configured to detect at least one characteristic in the image input data, for example, an abnormal tissue or a normal tissue, or the shape of one or more features. For example, the image input data can indicate or represent an organic tissue, for example, another sample of an organic tissue (different from the sample of the organic tissue described above).

[0039] According to at least some embodiments, the system can be configured to overlay the image input data with a visual overlay representing at least one characteristic of a sample of an organic tissue, for example, by highlighting an abnormal tissue or a normal tissue, or by highlighting the shape of one or more features.

[0040] One or more interfaces 130 can correspond to one or more inputs and / or outputs for receiving and / or transmitting information that can be digital (bit) values by specific code, within a module, between modules, or between modules of various entities. For example, one or more interfaces 130 can include interface circuits configured to receive and / or transmit information.

[0041] According to an embodiment, one or more processors 120 can be implemented using any means for processing, such as one or more processing units, one or more processing devices, a processor, a computer, or a programmable hardware component operable with correspondingly adapted software. In other words, the above-described functions of one or more processors 120 can also be implemented as software, and in such a case, this software is executed in one or more programmable hardware components. Such hardware components can include general-purpose processors, digital signal processors (DSPs), microcontrollers, and the like.

[0042] According to at least some embodiments, one or more storage modules 110 can include at least one element from a group of computer-readable storage media, such as magnetic storage media or optical storage media, for example, hard disk drives, flash memories, floppy disks, random access memories (RAM), programmable read-only memories (PROM), erasable programmable read-only memories (EPROM), electrically erasable programmable read-only memories (EEPROM), or network storage.

[0043] Further details and aspects of the embodiments are referred to in relation to the proposed concept or one or more of the examples described above or below. An embodiment can include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the examples described above or below.

[0044] FIG. 2a shows a flowchart of one embodiment of a corresponding method (implemented by a computer) for training a machine learning model. This method includes step 210 of obtaining a plurality of images of a sample of an organic tissue. The plurality of images are taken using a plurality of different imaging characteristics. This method includes step 220 of training a machine learning model using the plurality of images. The plurality of images used as training samples and information regarding at least one characteristic of the sample of the organic tissue are used as the desired output of the machine learning model. The machine learning model is trained such that the machine learning model is suitable for detecting at least one characteristic of the sample of the organic tissue in image input data that reproduces an appropriate subset of the plurality of different imaging characteristics. This method includes step 230 of providing the machine learning model. Optionally, this method includes step 250 of using the machine learning model with image input data that reproduces an appropriate subset of the plurality of different imaging characteristics to detect at least one characteristic of the sample of the organic tissue.

[0045] Alternatively, this detection can be performed separately from the training of the machine learning model. Thus, the machine learning model can be used within a microscope system while the training is being performed in a different computer system. Thus, FIG. 2b shows a flowchart of one embodiment of a method for detecting at least one characteristic of a sample of an organic tissue. Optionally, this method includes step 240 of obtaining a machine learning model and / or image input data that reproduces an appropriate subset of the plurality of different imaging characteristics. This method includes step 250 of using the machine learning model with image input data that reproduces an appropriate subset of the plurality of different imaging characteristics to detect at least one characteristic of the sample of the organic tissue, for example.

[0046] Further details and aspects of the embodiments are referred to in relation to the proposed concepts or one or more of the examples described above or below. An embodiment can include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the examples described above or below.

[0047] FIG. 3 shows a block diagram of a microscope system 300. For example, the microscope system can be configured to implement at least one of the methods of FIGS. 2a and / or 2b, and / or this microscope system can include the system of FIG. 1. Accordingly, some embodiments relate to microscopes that include a system as described in relation to one or more of FIGS. 1 through 2b. Alternatively, the microscope may be part of a system as described in relation to one or more of FIGS. 1 through 2b, or may be connected to a system as described in relation to one or more of FIGS. 1 through 2b. FIG. 3 shows a schematic diagram of a microscope system 300 configured to implement the methods described herein. System 300 includes a microscope 310 and a computer system 320. Microscope 310 is configured to capture images and is connected to computer system 320. Computer system 320 is configured to implement at least a portion of the methods described herein. Computer system 320 may be configured to execute a machine learning algorithm. Computer system 320 and microscope 310 may be separate entities, or may be integrated within a common housing. Computer system 320 may be part of the central processing system of microscope 310, and / or computer system 320 may be part of a sub-component of microscope 310, such as a sensor, actuator, camera, or illumination unit of microscope 310.

[0048] The computer system 320 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system distributed across various locations such as local clients and / or one or more remote server farms and / or data centers, comprising one or more processors and one or more storage devices). The computer system 320 may include any circuit or combination of circuits. In one embodiment, the computer system 320 may include one or more processors, which can be of any kind. As used herein, a processor may contemplate any kind of computing circuit, such as, for example, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other kind of processor or processing circuit, but is not limited thereto. Other kinds of circuits that may be included in the computer system 320 may be custom circuits, application specific integrated circuits (ASICs), etc., for example, one or more circuits (such as communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 320 may include one or more storage devices that may include one or more storage elements suitable for specific applications, such as main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives for handling removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 320 may include a display device, one or more speakers and a keyboard and / or a mouse, a trackball, a touch screen, a controller that may include a voice recognition device, or any other device that enables a user of the system to input information into the computer system 320 and receive information from the computer system 320.

[0049] Some or all of the steps may be performed by a hardware device (or using a hardware device) such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the critically important steps may be performed by such a device.

[0050] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is executable by a non-transitory recording medium, which is a digital recording medium such as an electronically readable control signal stored therein that cooperates (or is capable of cooperating) with a programmable computer system to implement each method, such as, for example, a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, and an EPROM, an EEPROM, or a FLASH memory. Thus, the digital recording medium may be computer-readable.

[0051] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system so that any of the methods described herein are implemented.

[0052] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which code is operative to implement any of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0053] Another embodiment includes a computer program stored on a machine-readable carrier for implementing any of the methods described herein.

[0054] Thus, in other words, embodiments of the present invention are computer programs having program code for implementing any of the methods described herein when the computer program is executed on a computer.

[0055] Accordingly, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) including a stored computer program for implementing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein including a processor and a recording medium.

[0056] Thus, another embodiment of the present invention is a data stream or signal sequence representing a computer program for implementing any of the methods described herein. The data stream or signal sequence may be configured to be transferred, for example, via a data communication connection such as the Internet.

[0057] Another embodiment includes a processing means, for example, a computer or a programmable logic device configured or adapted to implement any of the methods described herein.

[0058] Another embodiment includes a computer having an installed computer program for implementing any of the methods described herein.

[0059] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0060] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to implement any of the methods described herein. Generally, and advantageously, the methods are implemented by any hardware device.

[0061] Further details and aspects of the embodiments are referred to in relation to the proposed concept or one or more of the foregoing or following examples. The embodiments can include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the foregoing or following examples.

[0062] At least some embodiments relate to the use of artificial intelligence (AI), in the form of, for example, a machine learning model, for interpreting microscope-captured images, for example in a surgical microscope. For example, (all) available information can be collected and AI can be used to increase diagnostic information. According to an embodiment, artificial intelligence is used to interpret captured images in a surgical microscope.

[0063] If one attempts to use some microscope systems together with AI, the limitations of those microscope systems are the collection of images for both the training and application of AI. In particular, since microscopes acquire various images sequentially, it may be even more difficult and cumbersome to collect various data. Even more difficult may be the data annotation step, i.e., the step of using a specialized surgeon to annotate normal and abnormal images, or even more difficult may be the manual segmentation of normal tissue regions and diseased tissue regions.

[0064] The inventors propose a method to make the training and application of AI in surgical microscopes easier, more efficient, and more accurate. According to at least some embodiments, the microscope can capture a large number of images (e.g., multiple images) of reflection and fluorescence simultaneously in real time (e.g., substantially simultaneously). In other words, the camera can capture, for each frame, a plurality of images, and according to some embodiments, up to three reflection spectral images and three fluorescence spectral images. This number may increase in the future. By being able to capture images simultaneously and instantaneously, the images can be correlated on a pixel-by-pixel basis. These multiple images that can be correlated on a pixel-by-pixel basis can make more data available for neural network correlation, thus providing an excellent platform for AI, i.e., for training a machine learning model. As a result, embodiments can be based on using multiple images captured in various spectral bands in reflection and fluorescence by using external fluorescent dyes such as fluorescein, indocyanine green (ICG), and 5-ALA, or tissue autofluorescence (without using a fluorescent dye), and according to embodiments, it is possible to attempt to train the system to detect various abnormal and / or normal tissues. A specific case where the use of AI is relatively easy and not obvious is to train the system (i.e., the machine learning model) using 5-ALA-induced fluorescence images and detect brain tumors from non-fluorescent images. Specifically, 5-ALA emits fluorescence with relatively good sensitivity and specificity in brain tumor tissue and is thus used for intraoperative guidance in brain tumor resection. In other words, it is quite easy to segment the tumor area from the fluorescence image just by setting a fluorescence intensity threshold.

[0065] Even if security and reliability may be added by expert review, it can be completely automated by a computer without the need for human intervention, thereby automatically annotating the captured images (e.g., to obtain information about abnormal or normal tissues). The ability of the system to simultaneously capture white light reflection images (color images) allows for real-time data collection throughout the entire duration of such a surgical procedure. This can eliminate the time-consuming and costly process of capturing and annotating images. The goal of this AI / machine learning training would be to attempt to infer the presence of tumors in the brain by simply looking at the tissue without administering 5-ALA, which is costly and not always available for economic or regulatory reasons.

[0066] Further details and aspects of the embodiments are referred to in relation to the proposed concept or one or more of the examples described above or below. An embodiment can include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more of the examples described above or below.

[0067] Figures 4a to 6b show schematic diagrams for the detection of at least one characteristic of a sample of an organic tissue. Figures 4a and 4b show a sample of an organic tissue. In this case, the shape of the feature of the sample of the organic tissue can be visually recognized, but it is not clearly distinguishable in the first image (Figure 4a) taken with the first imaging characteristic (for example, white light reflection imaging). However, it is clearly visible in the second image (Figure 4b) taken with the second imaging characteristic (for example, using fluorescence imaging and the fluorescent dye has penetrated into the feature of the sample of the organic tissue). In such a case, the second image can be used to determine the shape of the feature of the sample of the organic tissue, and this can be used to generate a desired output for the training of a machine learning model. Therefore, the machine learning model can be used to determine the shape of the feature using only the first image.

[0068] Figures 5a to 5c show a similar situation. Here, two different features (a blood vessel shown as a line and a part of the tissue having a predetermined characteristic shown as a dot) can be visually recognized in the first image (Figure 5a) taken with the first imaging characteristic (for example, white light reflection imaging). In the second image (Figure 5b), the shape of a part of the tissue can be clearly visually recognized. For example, the second image can be taken using fluorescence imaging. Thus, using the second image, information regarding at least one characteristic of the sample of the organic tissue can be generated. Therefore, the machine learning model can be trained so that it is suitable for the machine learning model to identify the shape of a part of the sample of the organic tissue from only the first image. Further, the shape can be overlaid on the first image to generate a third image (see Figure 5c).

[0069] Similar examples are shown in Figures 6a and 6b. By using a machine learning model (that is, artificial intelligence), an annotation for showing the shape 600 of a certain region of a sample of an organic tissue can be added to the first image (Figure 6a) which can be white light reflection imaging taken by a camera (Figure 6a).

[0070] Further details and aspects of the embodiments are referred to in connection with the proposed concepts or one or more of the foregoing or following examples (e.g., FIGS. 1 - 3). Embodiments can include one or more additional optional features corresponding to one or more aspects of the proposed concepts or one or more of the foregoing or following examples.

[0071] Embodiments may be based on the use of a machine learning model or algorithm. Machine learning may refer to algorithms and statistical models that a computer system can use to perform a specific task without using explicit instructions, instead of relying on models and inferences. For example, in machine learning, instead of rule - based data transformation, data transformation inferred from the analysis of past data and / or training data may be used. For example, image content may be analyzed using a machine learning model or a machine learning algorithm. To analyze image content, a machine learning model may be trained using training images as input and training content information as output. By training a machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and related training content information (e.g., labels or annotations), the machine learning model "learns" to recognize image content, so that image content not included in the training data can be recognized using the machine learning model. The same principle may be used for other types of sensor data in the same way: by training a machine learning model with training sensor data and a desired output, the machine learning model "learns" the conversion between sensor data and the output, which can be used to provide an output based on non - training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be pre - processed to obtain a feature vector used as input to the machine learning model.

[0072] The machine learning model may be trained using training input data. The above example uses a training method called "supervised learning". In supervised learning, the machine learning model is trained using a plurality of training samples, where each sample may include a plurality of input data values and a plurality of desired output values, that is, each training sample is associated with a desired output value. By specifying both the training sample and the desired output value, the machine learning model "learns" during training what output value to provide based on input samples similar to the provided samples. In addition to supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack the corresponding desired output values. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm, or a similarity learning algorithm). When the output is restricted to a limited set of values (categorical variables), that is, when the input is classified into one of a limited set of values, a classification algorithm may be used. When the output may have any numerical value (within a range), a regression algorithm may be used. The similarity learning algorithm may be similar to both the classification algorithm and the regression algorithm, but is based on learning from examples using a similarity function that measures how similar or related two objects are. In addition to supervised learning or semi-supervised learning, unsupervised learning may be used to train the machine learning model. In unsupervised learning, only the input data may be supplied, and the unsupervised learning algorithm may be used to find structure in the input data (e.g., by grouping or clustering the input data, or by finding commonalities in the data). Clustering is the assignment of input data containing a plurality of input values to a plurality of subsets (clusters), so that the input values within the same cluster are similar according to one or more (predetermined) similarity criteria, but not similar to the input values contained in another cluster.

[0073] Reinforcement learning is the third group of machine learning algorithms. In other words, reinforcement learning may be used to train a machine learning model. In reinforcement learning, one or more software actors (referred to as "software agents") are trained to act in their surroundings. Based on the actions taken, a reward is calculated. Reinforcement learning is based on training one or more software agents to select actions such that (as revealed by an increase in reward) the cumulative reward increases and a software agent that performs better on a given task is obtained.

[0074] Furthermore, several techniques may be applied to part of the machine learning algorithm. For example, feature representation learning may be used. In other words, the machine learning model may be trained at least in part using feature representation learning, and / or the machine learning algorithm may include a feature representation learning component. Feature representation learning algorithms, which may be referred to as representation learning algorithms, not only store information in their inputs, but may also usefully transform the information, often as a preprocessing step before performing classification or prediction. Feature representation learning may be based on, for example, principal component analysis or cluster analysis.

[0075] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide the identification of input values that raise suspicion by being significantly different from most of the input or training data. In other words, the machine learning model may be trained at least in part using anomaly detection, and / or the machine learning algorithm may include an anomaly detection component.

[0076] In some examples, the machine learning algorithm may use a decision tree as a prediction model. In other words, the machine learning model may be based on a decision tree. In a decision tree, an observation regarding an item (e.g., a set of input values) may be represented by a branch of the decision tree, and an output value corresponding to this item may be represented by a leaf of the decision tree. The decision tree may support both discrete and continuous values as output values. When discrete values are used, the decision tree may be represented as a classification tree, and when continuous values are used, the decision tree may be represented as a regression tree.

[0077] Correlation rules are another technique that can be used in machine learning algorithms. In other words, the machine learning model may be based on one or more correlation rules. Correlation rules are created by identifying relationships between variables in large amounts of data. The machine learning algorithm may identify and / or utilize one or more correlative rules that represent knowledge derived from the data. These rules may be used, for example, to store, manipulate, or apply knowledge.

[0078] Machine learning algorithms typically are based on a machine learning model. In other words, the term "machine learning algorithm" may represent a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may represent a data structure and / or a set of rules that represent learned knowledge (e.g., based on training performed by a machine learning algorithm). In an embodiment, the usage of a machine learning algorithm may mean the usage of one underlying machine learning model (or multiple underlying machine learning models). The usage of a machine learning model may mean that the machine learning model and / or the set of data structures / rules that are the machine learning model are trained by a machine learning algorithm.

[0079] For example, the machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or brain. An ANN includes a plurality of interconnected nodes and a plurality of junctions between the nodes, so-called edges. Typically, there are three types of nodes, namely input nodes that receive input values, hidden nodes that are connected to other nodes (only), and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information from one node to another. The output of a node may be defined as a (non-linear) function of its input (e.g., the sum of its inputs). The input of a node may be used in a function based on the "weights" of the edges or nodes that provide the input. The weights of the nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may include adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input.

[0080] Alternatively, the machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing an input with a plurality of training input values belonging to one of two categories. A support vector machine may be trained to assign new input values to one of two categories. Alternatively, the machine learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may represent a set of probabilistic variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machine learning model may be based on a genetic algorithm, which is a heuristic method that mimics a search algorithm and the process of natural selection.

[0081] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0082] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent descriptions of corresponding methods, where a block or apparatus corresponds to a step or a feature of a step. Similarly, aspects described in the context of a step also represent descriptions of corresponding blocks or items or features of a corresponding apparatus. Some or all of a step may be performed by a hardware device (or using a hardware device) such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the extremely important steps may be performed by such a device.

Description of Reference Numerals

[0083] 100 System 110 One or more storage modules 120 One or more processors 130 One or more interfaces 210 Step of acquiring a plurality of images of a sample of an organic tissue 220 Step of training a machine learning model 230 Step of supplying a machine learning model 240 Step of obtaining a machine learning model 250 Step of using a machine learning model 300 Microscope system 310 Microscope 320 Computer system 600 Region of a sample of an organic tissue

Claims

1. A system (100) comprising one or more storage modules (110) and one or more processors (120), wherein the system (100) acquires a plurality of images of a sample of an organic tissue taken using a plurality of different imaging characteristics, is configured to train a machine learning model using the plurality of images, the plurality of images being used as training samples, information about at least one characteristic of the sample of the organic tissue being used as a desired output of the machine learning model, and the machine learning model being suitable for detecting the at least one characteristic of the sample of the organic tissue in image input data that reproduces an appropriate subset of the plurality of different imaging characteristics, the system (100) is configured to supply the machine learning model, the system (100) is further configured to train the machine learning model such that the machine learning model identifies the shape of a part of the sample of the organic tissue from only the first image among the plurality of images, System (100).

2. The information about the at least one characteristic of the sample of the organic tissue represents at least one part of the sample of a normal or abnormal organic tissue, and / or the information about the at least one characteristic of the sample of the organic tissue represents the shape of one or more features of the sample of the organic tissue, The system according to claim 1.

3. Information about abnormal or normal tissue is used as a desired output of the training of the machine learning model, the machine learning model is trained such that the machine learning model is suitable for detecting abnormal or normal tissue in image input data that reproduces an appropriate subset of the plurality of different imaging characteristics, The system according to claim 1 or 2.

4. Information about the shape of one or more features of the sample of the organic tissue is used as a desired output of the training of the machine learning model, the machine learning model is trained such that the machine learning model is suitable for determining the shape of the one or more features in image input data that reproduces an appropriate subset of the plurality of different imaging characteristics, The system according to any one of claims 1 to 3.

5. The plurality of various imaging characteristics relate to at least one of various spectral bands, various imaging modes, various polarizations, and various time points in a time-resolved imaging series. The system according to any one of claims 1 to 4.

6. The plurality of images include one or more elements of a group consisting of microscopic images taken in various spectral bands, microscopic images taken in various imaging modes, microscopic images taken in various polarizations, and microscopic images representing various time points in a time-resolved imaging series. The system according to any one of claims 1 to 5.

7. The plurality of images include one or more three-dimensional representations of the sample of the organic tissue, and / or the information regarding the at least one characteristic of the sample of the organic tissue is based on the three-dimensional representation of the sample of the organic tissue. The system according to any one of claims 1 to 6.

8. The information regarding the at least one characteristic of the sample of the organic tissue is based on one of the plurality of images, and the system is configured to process the one image in order to obtain the information regarding the at least one characteristic of the sample of the organic tissue. The system according to any one of claims 1 to 7.

9. The one image is taken using an imaging characteristic representing a specific type of abnormal tissue, and / or the one image is taken using an imaging characteristic representing the shape of one or more features of the sample of the organic tissue, and / or the one image is a fluorescence spectrum image, and / or the one image has been excluded as a training sample. The system according to claim 8.

10. The information regarding the at least one characteristic of the sample of the organic tissue is based on two or more of the plurality of images, and each of the two or more images is taken using an imaging characteristic representing a specific type of abnormal tissue, or each of the two or more images is taken using an imaging characteristic representing the shape of one or more features of the sample of the organic tissue. The system according to claim 8 or 9.

11. At least one subset of the plurality of images reproduces a spectral band adjusted to at least one external fluorescent dye applied to the sample of the organic tissue, and / or at least one subset of the plurality of images reproduces a spectral band adjusted to the autofluorescence of at least a part of the sample of the organic tissue, The system according to any one of claims 1 to 10.

12. The system is configured to correlate the plurality of images on a pixel-by-pixel basis, The machine learning model is trained based on the correlated plurality of images, The system according to any one of claims 1 to 11.

13. The plurality of images includes one or more reflection spectral images and one or more fluorescence spectral images, The one or more reflection spectral images reproduce the visible light spectrum, and / or each of the one or more fluorescence spectral images reproduces a spectral band adjusted to fluorescence at a specific wavelength observable in the sample of the organic tissue, The system according to any one of claims 1 to 11.

14. The system is configured to use the machine learning model together with image input data that reproduces an appropriate subset of the plurality of various imaging characteristics in order to detect the at least one characteristic of the sample of the organic tissue in the image input data. The system according to any one of claims 1 to 13.

15. The image input data is image input data of a camera operating within the visible light spectrum, and / or the image input data is obtained from tissue that has not been treated with an external fluorescent dye, The system according to claim 14.

16. A method for training a machine learning model, the method comprising: a step (210) of obtaining a plurality of images of a sample of an organic tissue taken using a plurality of various imaging characteristics, A step (220) of training a machine learning model using the plurality of images, wherein the plurality of images are used as training samples, information regarding at least one characteristic of the sample of the organic tissue is used as a desired output of the machine learning model, and the machine learning model is suitable for detecting the at least one characteristic of the sample of the organic tissue in image input data that reproduces an appropriate subset of the plurality of various imaging characteristics (220); A step (230) of supplying the machine learning model; comprising; The step of training (220) includes the step of training the machine learning model such that the machine learning model identifies a partial shape of the sample of the organic tissue from only a first image of the plurality of images. Method. **Claim 17** The method further includes a step (250) of using the machine learning model with image input data that reproduces an appropriate subset of the plurality of various imaging characteristics. The method according to claim 16. **Claim 18** A computer program including program code for implementing at least one of the methods according to any one of claims 16 or 17 when the computer program is executed by a processor. **Claim 19** A microscope system (300) configured to execute the method according to claim 16 or 17.

Images