Providing similar medical image

Abstract

To provide a method for providing a similar medical image, an image providing system, a program, and a storage medium.SOLUTION: In a method, a feature unit ML-F receives a first medical image IMG-1 for a first patient and applies a first machine learning model ML-1 to the first medical image to determine a first feature vector FV-1. The first machine learning model has been trained on training data sets, each training data set including training medical images and associated non-imaging data. The method also receives a plurality of second feature vectors. The plurality of second feature vectors is a result of applying the first machine learning model to the plurality of second medical images. The method further determines a similar feature vector from the plurality of second feature vectors based on the first feature vector and selects a similar medical image from the plurality of second medical images. Similar medical images are associated with similar feature vectors.SELECTED DRAWING: Figure 4

Description

[Background technology]

[0001] Regardless of grammatical usage of the term, individuals of male, female, or other gender identity are included within each term.

[0002] Finding similar patients is an important topic in medical technology, particularly in the context of cancer diagnosis and treatment planning, especially when faced with rare or unusual cases. For example, identifying patients with comparable molecular structures and tumor characteristics is important to guide medical professionals to effective treatment options and enhance clinical judgment. In such complex cases, standard protocols may not be sufficient, and medical professionals may suffer from uncertainty about the best course of action. Specifically, it would be preferable to find similar patients based on comparing respective medical images, such as histopathology slide images.

[0003] Traditionally, pathologists have primarily relied on manual inspection of histopathology slides under a microscope to identify and compare patterns and features within tumor areas. The findings are usually discussed in tumor board meetings, including medical experts from various disciplines, where additional data and modalities are considered. Tumor boards can also be used to identify similar patients. However, this approach is labor-intensive, time-consuming, and prone to inter- and intra-observer variability, which can lead to discrepancies in patient evaluation and treatment decisions.

[0004] Content-based image retrieval systems have been employed to find similar images in medical databases using global features such as color, texture, and shape. Although such systems have proven useful in certain cases, they are not suited to efficiently match patients based on local patterns and features within the tumor region and lack data integration related to other factors to guide the search process. In digital pathology, these systems are primarily based on visual features and do not incorporate other data, which may result in search images that are visually similar to each other but diagnostically represent different cancers. Summary of the Invention

[0005] It is an object of the present invention to improve detection of similar patients based on medical images. This problem is solved according to the independent claims. Further advantageous embodiments and further advantageous features are described in the dependent claims and in the description.

[0006] The solution according to the invention will be described below with respect to the systems according to the claims and the methods according to the claims. Features, advantages or alternative aspects herein may also be applied to the corresponding subject matter of the other claims and vice versa. That is, the system may be improved using features that are described or claimed in terms of the corresponding method. In such cases, the functional features of the method are implemented by the object units of the system.

[0007] Further, in the following, the solution according to the present invention will be described with respect to a method and system for providing similar medical images and with respect to a method and system for providing a trained machine learning model. Features, advantages, or alternative aspects herein may also be applied to the subject matter of other claims, and vice versa. That is, each claim and aspect for providing a trained machine learning model may be improved with features described or claimed in terms of providing similar medical images. In particular, the datasets used in the method and system for providing similar medical images may have the same characteristics and features as the corresponding datasets used in the method and system for providing a trained machine learning model, and the trained machine learning models provided by these methods and systems may also be used in the method and system for providing similar medical images.

[0008] In the following, "particularly" is used to indicate optional and / or beneficial additional features. Furthermore, the expressions "applying a model to data" or "applying a unit to data" are used to indicate that the data are used as input data to a model or unit, or that input data including the data (and possibly other data) are used.

[0009] In a first aspect, the present invention relates to a computer-implemented method for providing similar medical images. The method includes receiving a first medical image of a first patient. The method further includes determining a first feature vector by applying a first machine learning model to the first medical image. The first machine learning model is trained based on a training dataset, each of the training datasets including training medical images and associated non-imaging data. The method further includes receiving a plurality of second feature vectors. The plurality of second feature vectors is a result of applying the first machine learning model to the plurality of second medical images. The method further includes determining a similar feature vector from the plurality of second feature vectors based on the first feature vector. The method further includes selecting a similar medical image from the plurality of second medical images, the similar medical image being associated with the similar feature vector. The method further includes providing the similar medical images. In particular, providing the similar medical images may include storing, transmitting, and / or displaying the similar medical images.

[0010] Specifically, the steps of receiving a first medical image and receiving a plurality of second feature vectors are performed by an input interface, in particular an input interface of an image providing system. Specifically, the steps of determining the first feature vector, determining similar feature vectors based on the first feature vector, and selecting similar medical images are performed by a calculation unit, in particular a calculation unit of the image providing system. Specifically, the step of providing similar medical images is performed by an output interface, in particular an output interface of the image providing system.

[0011] Specifically, a medical image is an image of a patient recorded by a medical imaging modality. A medical image can be an X-ray image, a computed tomography image (acronym "CT image"), a magnetic resonance image (acronym "MR image"), a positron emission tomography image (acronym "PET image"), a single photon emission computed tomography image (acronym "SPECT image"), and / or an ultrasound image (acronym "US image"). A medical image can also be a microscopic image, a histopathology image, and / or a time-series biosignal analysis image. In particular, a medical image can be a whole slide image (WSI).

[0012] Specifically, a medical image can be a two-dimensional image, a three-dimensional image, or a four-dimensional image, in which there are three spatial dimensions and one time dimension in a four-dimensional image.

[0013] A medical image may include a large number of pixels or voxels. "Pixel" is used primarily for a two-dimensional image building block, and "voxel" is used for an image building block with any dimension (primarily three or more dimensions). However, in the following, "pixel" may be used synonymously with "pixel and / or voxel." Each pixel may include at least one intensity value (e.g., Hounsfield units in the case of CT images) that corresponds to a particular tissue property.

[0014] A medical image may correspond to or be encapsulated in one or more DICOM files. It should be understood that when DICOM is referred to herein, it refers to the "Digital Imaging and Communications in Medicine" (DICOM) standard, e.g., according to the current DICOM PS3.1 2020c standard (or any later or earlier version of that standard). It is also possible that several medical images are encapsulated in one DICOM file.

[0015] In particular, the medical image may include additional metadata, such as a patient identifier, the patient's name, an identifier of the modality used to acquire the medical image, the date and time of image acquisition, the imaging protocol used to acquire the medical image, and / or other metadata associated with the medical image and / or its acquisition process.

[0016] Specifically, the first medical image, the second medical image, the similar medical image, and the training medical image are the same type of medical image, i.e., the first medical image, the second medical image, the similar medical image, and the training medical image are recorded with the same type of imaging modality (e.g., all medical images are recorded by CT, or all medical images are whole slide images). Specifically, the first medical image, the second medical image, the similar medical image, and the training medical image may have the same dimensions. Specifically, the first medical image, the second medical image, the similar medical image, and the training medical image may be the same size (e.g., measured in number of pixels) along each of their dimensions. Specifically, the set of second medical images may include the set of training medical images and / or the set of second medical images may be consistent with the set of training medical images.

[0017] In general, machine learning models mimic the cognitive functions that humans use to interact with other minds. Specifically, by training on training data, machine learning models can adapt to new situations and detect and extrapolate patterns. In other words, a "machine learning model" is a "trained function."

[0018] Generally speaking, the parameters of the machine learning model can be adapted by training. Specifically, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (also called "feature learning") can also be used. Specifically, the parameters of the machine learning model can be iteratively adapted by several training steps. Specifically, during training, a certain cost function can be minimized. Specifically, during training of the neural network, the backpropagation algorithm can be used.

[0019] Specifically, the machine learning model can be composed of a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or the machine learning model can be based on k-means clustering, Q-learning, a genetic algorithm, and / or an association rule. In particular, the neural network can be a deep neural network, a convolutional neural network, or a deep convolutional neural network. Furthermore, the neural network can be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0020] In the context of the present disclosure, the phrase "applying a machine learning model to a medical image" can be interpreted as there may be additional data along with the medical image that can be used as input to the machine learning model. Also, in the context of the present disclosure, the phrase "applying a machine learning model to a medical image" can be interpreted as at least a portion of the medical image being used as input to the machine learning model (possibly with additional data). In other words, the phrase "applying a machine learning model to a medical image" can be interpreted as "applying a machine learning model to input data, the input data consisting of at least a portion of the medical image."

[0021] In general, a feature is an individual measurable quality or characteristic of a phenomenon. In the context of this disclosure, a feature can be an individual measurable quality or characteristic associated with a medical image. Specifically, a feature vector is an n-dimensional vector of numerical features that represent the same object (the vector space associated with the vector can be called a "feature space"). In the context of this disclosure, a feature vector can be an n-dimensional vector of numerical features that represent a medical image. In the context of this disclosure, any ordered set of numerical values ​​can be interpreted as a feature vector.

[0022] Specifically, the first feature vector, the similar feature vector, and the second feature vector may have the same dimension. In other words, the first feature vector, the similar feature vector, and the second feature vector may be elements of the same feature space.

[0023] Specifically, the non-imaging data is medical data. Specifically, the non-imaging data is medical data relating to a patient. Specifically, the non-imaging data is medical data that does not include medical images, although the non-imaging data may be derived from medical images (e.g., a diagnosis derived by a radiologist or pathologist based on a medical image). Specifically, the non-imaging data may include and / or be mapped to at least one numerical value. Specifically, the non-imaging data may be an m-dimensional vector, where each value of the m-dimensional vector relates to a particular patient.

[0024] The inventors of the present invention have found that similar images with higher relevance can be determined by using the feature vector generated by the first machine learning model trained on a set of imaging data and non-imaging data to search for similar images or similar patients. This is based on the fact that the parameters of the first machine learning model during training can incorporate non-imaging data, so that the feature vector extracted by the first machine learning model is configured (by the training process) to predict not only the visual impression of the medical image, but also each non-imaging data of the same patient.

[0025] Specifically, when using the proposed method to find similar images or similar patients for a particular disease (e.g., certain cancers), the first machine learning model, based on training according to the present proposal, becomes adept at identifying diagnostically relevant morphological features and patterns and can accurately search for patients whose diseases show similarities in both morphological and non-imaging properties. This advance provides significant clinical benefit and utility as it enables medical professionals to determine personalized treatment options, enhance diagnostic accuracy, and ultimately improve patient outcomes, especially in rare and complex diseases.

[0026] According to a further possible aspect of the present invention, each second feature vector of the plurality of second feature vectors is a result of applying the first machine learning model to only one second medical image of the plurality of second medical images, i.e., there may be a one-to-one correspondence between the plurality of second feature vectors and the plurality of second medical images.

[0027] According to a further possible embodiment of the present invention, the non-imaging data has a different structure from the first feature vector. Specifically, the non-imaging data has a lower dimension than the first feature vector. Specifically, when both the non-imaging data and the first feature vector can be interpreted as vectors of numerical values, the non-imaging data has a lower dimension than the first feature vector if the dimension of the vector representing the non-imaging data is smaller than the dimension of the feature vector or the vector representing the feature vector.

[0028] According to a further aspect of the invention, the first medical image is a histopathology slide image, in particular a whole slide image.

[0029] The histopathology slide image may specifically be a two-dimensional pixel image. The histopathology slide image may map at least a portion of a tissue slide prepared from a patient's tissue sample. Preparation of the tissue slide from the tissue sample may include cutting out sections / sections (also called "punch biopsies" or "blocks") from the tissue sample (e.g., using a punch tool), which sections are cut into micrometer-thick slices, i.e., tissue slides. Preparation of the tissue slide may further include staining the tissue slide with a histopathological stain. The staining in this case serves to highlight different structures in the tissue slide, such as cell walls and cell nuclei, or to check medical indications, such as cell proliferation levels. In this process, separate histopathological stains may be used for separate issues.

[0030] A tissue slide stained with a first histopathological stain is or has been digitized or scanned to generate one or more histopathological slide images. For this purpose, the first tissue slide is imaged using a suitable digitizing station, for example known as a whole slide scanner. This station preferably scans the entire tissue slide mounted on the object carrier and converts it into a pixel image. In order to obtain the staining effect through the histopathological stain, the pixel image is preferably a color pixel image. Since in diagnosis both the overall impression of the tissue and the finely resolved cellular structure are important, the histopathological slide images usually have a very high pixel resolution. The histopathological slide images can then be digitally processed and, in particular, archived in a suitable database. When viewed microscopically, the histopathological slide images show the micro-tissue structure of the tissue sample, in particular the cellular structures and cells contained in the tissue sample. When viewed on a larger scale, the histopathological slide images show higher-level tissue structures and features, such as density and cellular morphologically continuous areas, such as tumor and stromal areas.

[0031] Specifically, the first medical image is a histopathology slide image depicting a slice of a patient's tissue stained with a histopathological stain, which may be an H&E (acronym for "hematoxylin and eosin") stain or an IHC (acronym for "immunohistochemistry") stain.

[0032] In the case of H&E staining, hematoxylin stains the cell nuclei and eosin stains the extracellular matrix and cytoplasm. H&E staining is the most widely used stain in digital pathology, making the present method of providing similar images widely applicable.

[0033] IHC staining involves the process of selectively identifying antigens (proteins) in cells of tissue sections by utilizing the principle of antibodies binding specifically to antigens in living tissue, thereby highlighting structures that are inaccessible to other stains such as H&E and that may be highly relevant in terms of a particular disease in which certain similar patients should be searched for.

[0034] In particular, the IHC staining may include a biomarker (e.g. in the form of an antibody) adapted to target (cytoskeletal) keratin. Keratin forms part of the cytoskeleton of cells and determines the mechanical properties of the cell. For this reason, the abundance of keratin constitutes a good tumor marker, since the expression level of keratin varies considerably in tumor cells. Thus, the use of keratin IHC staining makes it possible to identify adenocarcinomas (neoplasms of epithelial tissue). In particular, the IHC staining includes a keratin biomarker that targets various keratin forms, such as CK-5, CK-8, CK-14, CK-18 ("CK" stands for "cytoskeletal keratin"). It should be noted that the IHC staining may include different or additional biomarkers, such as p63 and AMACR biomarkers.

[0035] A whole slide image is a histopathology image that maps an entire tissue slide that has been prepared from a patient's tissue sample. A whole slide image may have a size of at least 4000 x 4000 pixels, or at least 10000 x 10000 pixels, or at least 1E6 x 1E6 pixels.

[0036] Specifically, when the first medical image is a histopathological slide image, each of the training medical images, the plurality of second medical images, and / or the similar medical images is also a histopathological slide image. Specifically, when the first medical image is a whole slide image, each of the training medical images, the plurality of second medical images, and / or the similar medical images is also a whole slide image.

[0037] The inventors of the present invention have recognized that histopathology slide images or whole slide images can depict detailed morphological structures of cells that are dependent on specific diseases or other non-patient data. Based on such morphological structures, it is possible to provide similar medical images that are highly relevant with specific morphological structures indicative of specific diseases or other non-patient data.

[0038] According to further aspects of the invention, the non-imaging data includes at least one of the following categories: omics data; diagnostic data; medical history; and / or demographic data.

[0039] Generally speaking, "omics data" refers to analytical results related to a patient's biomolecules, generated by various "omics" technologies, such as genomics, transcriptomics, proteomics, and metabolomics. Biomolecules include genes, RNA transcripts, proteins, and / or metabolites. Omics data can provide comprehensive insight into functions and interactions within a biological system. The inventors of the present invention have confirmed that omics data can reveal specific details about a patient's biological expression that can be effectively used to determine suitable feature vectors for finding similar medical images. In particular, omics data describes the functional properties of diseases such as cancer, so that search moieties with similar omics structures have special diagnostic value.

[0040] Generally speaking, "diagnostic data" refers to data related to the identification and classification of a disease, disorder, and / or medical condition in a patient. The data encompasses various types of data, including medical records, diagnostic test results, patient symptoms, and clinical observations. The inventors of the present invention have determined that diagnostic data provides detailed, curated information about a patient and / or a patient's particular disease, such that the diagnostic data can be effectively used to determine feature vectors suitable for finding similar medical images.

[0041] Generally speaking, a "medical history" refers to a record of at least a subset of medical interventions and treatments that a patient has undergone for a particular disease, condition, or illness. Medical history data may include information such as the type of treatment administered, dosage, duration, treatment results, and adverse reactions or side effects experienced. The inventors of the present invention have determined that medical history provides valuable information about a patient's particular disease and its response to specific treatment options. The treatment data can be effectively used to determine feature vectors suitable for finding similar medical images, and may also provide hints about successful and unsuccessful treatments for similar patients.

[0042] Generally speaking, "demographic data" refers to a set of characteristics and information about a patient that provides insight into the patient's personal background, such as social, cultural, and economic background. Demographic data may include factors such as age, gender, race, and / or ethnicity, but may also include geographic location, education level, income, and occupation. The inventors of the present invention have determined that demographic data indicates similarities of particular patients and may be effectively used to determine feature vectors suitable for finding similar medical images.

[0043] According to a further aspect of the invention, the non-imaging data includes omics data, the omics data including at least one of genomic data, transcriptomic data, proteomic data, and metabolomic data.

[0044] Generally speaking, "genomic data" refers to at least a portion of a patient's set of genetic information or DNA (acronym for "deoxyribonucleic acid"). The data may include at least a portion of the patient's genome, including the sequence and composition of genes, alterations, and mutations within the DNA. The inventors of the present invention have determined that genomic data is indicative of a patient's genetic traits, susceptibility to disease, and response to drug therapy, and thus may be advantageously used to determine feature vectors suitable for finding similar medical images.

[0045] Generally speaking, "transcriptome data" refers to a set of information about the expression of genes in a particular tissue or cell type of a patient. The data may include the analysis of RNA molecules that are transcribed from genes and provide insight into which genes are active and produce proteins in a given biological sample of a patient. The inventors of the present invention have determined that transcriptome data indicates which genetic information of the data is actually expressed in a particular patient, and thus can be effectively used to determine feature vectors suitable for finding similar medical images.

[0046] Generally speaking, "proteomic data" refers to a set of information about proteins present in a particular tissue, cell, or biological sample of a patient. The data may include the types, amounts, modifications, and interactions of proteins in the patient's biological sample. The inventors of the present invention have determined that proteomic data provides valuable insights into the functional aspects of proteins, their roles in cellular processes, and their involvement in disease, and thus can be effectively used to determine feature vectors suitable for finding similar medical images.

[0047] Generally speaking, "metabolomic data" refers to information relating to molecules (known as metabolites) present in a patient's biological sample, such as blood, urine, or tissue. The data may include identification and quantification of metabolites and / or a global metabolic profile of the patient. The inventors of the present invention have determined that metabolomic data provides valuable information about metabolic states, disease markers, and responses to treatment associated with a patient, and thus may be advantageously used to determine feature vectors suitable for finding similar medical images.

[0048] According to a further possible aspect of the invention, the non-imaging data includes omics data and the training medical images represent a tumor of the patient, where the omics data is related to a type of cancer in the patient.

[0049] The inventors of the present invention have determined that omics data can provide detailed information about the specific nature of a patient's cancer and can be used to determine feature vectors suitable for finding similar medical images for the same patient type.

[0050] According to a further possible embodiment of the invention, the training medical images are histopathology slide images displaying the patient's tumor cells, in particular whole slide images displaying the patient's tumor cells.

[0051] The inventors of the present invention have determined that histopathology slide images representing a patient's tumor contain detailed information about the morphology of tumor cells and can be used to determine feature vectors suitable for finding similar medical images.

[0052] According to a further aspect of the invention, the first machine learning model is trained by training a second machine learning model that includes the first machine learning model, and the second machine learning model is trained to predict non-imaging data based on medical images, and advantageously the second machine learning model is a classifier.

[0053] According to a further possible embodiment of the present invention, the second machine learning model includes the first machine learning model and a classification unit, which processes the output of the first machine learning model and determines a classification value. In particular, the classification unit includes or consists of an additional neural network layer.

[0054] According to a further possible aspect of the present invention, at least one parameter of the second machine learning network is based on a difference between an output of the second machine learning network when the second machine learning network is applied to training medical images and non-imaging data corresponding to the training medical images.

[0055] According to a further aspect of the present invention, output data is generated by applying a first machine learning network to the medical images, the output data including a feature vector and non-imaging data, particularly where the feature vector and the non-imaging data are distinct.

[0056] The inventors of the present invention have determined that by using a first machine learning network that outputs both non-imaging data and feature vectors, the first machine learning network can be used both to determine similar images and to determine each non-imaging data.

[0057] According to a further aspect of the present invention, the first machine learning model includes a feature unit and an attention unit, and the attention unit is based on an attention mechanism. According to this aspect, the step of determining the first feature vector includes: determining a plurality of sub-features by applying the feature unit to the first medical image; and determining the first feature vector by applying the attention unit to the plurality of sub-features.

[0058] Specifically, each of the sub-features may be a vector having the same dimensions as the first feature vector.

[0059] Generally speaking, an attention mechanism is a subunit of a machine learning model that allows the machine learning model to focus on a particular part of the input of the attention mechanism by assigning different weights to each part of the input attention mechanism. This weighting is usually based on the relevance of each part of the input to the task performed by the machine learning model. In a neural network, the attention mechanism may correspond to at least one network layer. Specifically, the attention mechanism may refer to self-attention, dot product attention, scaled dot product attention, multi-head attention, and / or structured attention.

[0060] The inventors of the present invention have confirmed that the use of attention mechanisms in machine learning models can improve the accuracy of predictions of the machine learning models by enabling the machine learning models to pay attention to the most relevant information. The attention mechanisms can also make the machine learning models more efficient by only processing the most important data, reducing the required computational resources and making the machine learning models more scalable. The attention weights learned by the model can also provide insight into which parts of the data are most important, which can help improve the interpretability of the machine learning models.

[0061] According to a further aspect of the invention, the step of determining the first feature vector comprises reducing the dimensionality of the sub-features by principal component analysis and / or by applying a learnable bottleneck layer.

[0062] Generally speaking, principal component analysis is an orthogonal linear transformation that transforms input data into a new coordinate system, where the maximum variance from some scalar projection of the data is mapped to the first coordinate (called the first principal component), the second maximum variance on the second coordinate, and so on. Based on principal component analysis, the dimensionality can be reduced by only maintaining a certain number of the first coordinates. In general, a learnable bottleneck layer is a layer of a neural network with fewer output nodes than input nodes.

[0063] The inventors of the present invention have determined that by using dimensionality reduction techniques on feature vectors, the memory required to store the feature vectors (particularly those associated with a plurality of second medical images) can be reduced, and the computational power required to compare the feature vectors can be reduced, thereby enabling similar medical images to be provided more quickly and / or with greater accuracy.

[0064] According to a further aspect of the present invention, the first machine learning model is a Deep Local Feature Network. Deep Local Feature Networks and their advantages are described in the paper "H. Noh et al., "Large-Scale Image Retrieval with Attentive Deep Local Features", arXiv:1612.06321 (2016)", which is incorporated herein by reference.

[0065] According to a further aspect of the invention, the method includes determining a segmentation of a tumor in a first medical image, and in the step of determining a first feature vector, a machine learning model is applied to a portion of the first medical image that corresponds to the segmented tumor.

[0066] Specifically, the portion of the first medical image corresponding to the tumor may be a rectangular or square patch of the image. Specifically, the rectangular or square patch may include all of the segmented portion of the tumor, or the segmentation may include the entire rectangular or square patch. Alternatively, the rectangular or square patch and the tumor segmentation may overlap.

[0067] The inventors of the present invention have determined that using the portion of the medical image corresponding to the tumor as input for the machine learning model provides more accurate results in searching for similar patients since in many cases similar patients with cancer disease should be determined and imaging features related to the tumor are particularly indicative of the type of cancer disease.

[0068] According to a further aspect of the present invention, the method includes dividing a first medical image into a plurality of first patches, and in the step of determining a first feature vector, a first machine learning model is applied to one of the first patches. Specifically, the first machine learning model can be applied to the plurality of first patches to determine the respective first feature vectors. Specifically, the first machine learning model can be applied to all of the first patches to determine the first feature vectors for all of the first patches.

[0069] Specifically, each patch may be a rectangular or square patch. In particular, each patch may be mutually exclusive (such that for each pixel of the first medical image, there is only one patch that contains this pixel) and / or inclusive (such that for each pixel of the first medical image, there is at least one patch that contains this pixel). Specifically, all patches may have the same size in terms of number of pixels.

[0070] The inventors of the present invention have found that the first medical image often contains a huge number of pixels (especially for the whole slide image) that cannot be processed by the machine learning model with high accuracy. By dividing the first medical image into patches and then applying the machine learning model to the patches, the input data of the machine learning model can be reduced and better computational efficiency can be achieved. Furthermore, by calculating the feature vectors of the patches, it is possible to link similar patches that are related in the comparison images.

[0071] According to a further possible embodiment of the present invention, for each of the first patches, a corresponding first feature vector is determined by applying a first machine learning model to each of the first patches.The inventors of the present invention have confirmed that by calculating the first feature vector for each patch of the first medical image, similar medical images can be determined not based on the entire image, but on small parts that are more likely to be relevant.Furthermore, it may be possible to find similar medical images for each part of the first medical image.

[0072] According to a further possible aspect of the present invention, the first feature vector is based on the output of the first machine learning model when applied to each of the first patches from the subset of the first patches. Specifically, the first feature vector can be an average or weighted average of the output of the first machine learning model when applied to each of the first patches from the subset of the first patches. The inventors of the present invention have recognized that by calculating the first feature vector based on multiple patches, the overall impression of the first medical image can be included in the calculation of the first feature vector together with information from the entire first medical image, while not needing to process the entire first medical image at the same time, which increases computational efficiency.

[0073] According to a further possible embodiment of the present invention, the subset of the first patches is selected based on an attention mechanism of the first machine learning model. The inventors of the present invention have confirmed that by using the attention mechanism, the most relevant first patches will be included as a basis for calculating the first feature vector.

[0074] According to a further aspect of the invention, the similar medical images include a plurality of second patches, one of the second patches being associated with a similar feature vector, and providing the similar medical images includes displaying an association between one of the first patches and one of the second patches.

[0075] According to a further possible aspect of the present invention, providing similar medical images includes displaying an association between the first patch and the second patch based on a similarity of feature vectors associated with the first patch and the second patch, in particular, the association may be displayed if the similarity meets a predetermined threshold.

[0076] Generally speaking, the association between one of the first patches and one of the second patches is a (visual) indication that one of the first patches and one of the second patches are associated. Specifically, the association can be a visual indication that both one of the first patches and one of the second patches are marked in the same way (e.g., with the same border type or with the same symbol). Alternatively or additionally, the association can be a visual indication that connects one of the first patches and one of the second patches (e.g., by a line). Alternatively or additionally, the association can be a visual indication on the display based on the spatial relationship between one of the first patches and one of the second patches (e.g., displaying one of the first patches and one of the second patches side by side).

[0077] The inventors of the present invention have determined that by displaying associations, a user can easily identify associated patches in the first medical image and the similar medical images that led to the selection of the similar medical image.

[0078] In a second aspect, the present invention relates to a computer-implemented method for providing a first machine learning model. The method includes receiving a training dataset including training medical images and associated non-imaging data. The method further includes determining output data by applying a second machine learning model to the training medical images, the second machine learning model including the first machine learning model. The method further includes modifying parameters of the second machine learning model based on differences between the non-imaging data and the output data. The method further includes extracting a first machine learning model from the second machine learning model and providing the first machine learning model.

[0079] Specifically, the difference between the non-imaging data and the output data can be d-norm or mean square error (corresponding to the sum of squared differences of the entries in the vector). Specifically, the parameters of the second machine learning model can be modified based on reducing the difference between the non-imaging data and the output data, for example by using gradient descent. Specifically, when the second machine learning model is a neural network, a backpropagation algorithm can be used in the gradient descent. Specifically, the first machine learning model includes the parameters of the second machine learning model.

[0080] Specifically, extracting the first machine learning model from the second machine learning model (in this embodiment where the second machine learning model includes the first machine learning model) may correspond to cropping the second machine learning model to include only those portions that are also included in the first machine learning model. In the case where the first and second machine learning models are artificial neural networks, extracting the first machine learning model from the second machine learning model may correspond to using only those nodes and weights of the second machine learning model that are also included in the first machine learning model.

[0081] The inventors of the present invention have determined that a first machine learning model may be trained based on the above method for use in any of the methods according to the first aspect of the present invention.

[0082] According to a third aspect, the present invention relates to an image providing system for providing similar medical images, the system comprising: means for receiving a first medical image relating to a first patient; means for determining a first feature vector by applying a first machine learning model to a first medical image; the first machine learning model is trained based on training datasets, each of the training datasets including training medical images and associated non-imaging data; means for receiving a plurality of second feature vectors; the plurality of second feature vectors are a result of applying the first machine learning model to the plurality of second medical images; means for determining a similar feature vector from a plurality of second feature vectors based on the first feature vector; means for selecting a similar medical image from a plurality of second medical images; Similar medical images are associated with similar feature vectors, and a means for providing a similar medical image.

[0083] According to a fourth aspect, the present invention relates to an image providing system for providing similar medical images, the system comprising: Receiving a first medical image relating to a first patient; determining a first feature vector by applying a first machine learning model to a first medical image; the first machine learning model is trained based on training datasets, each of the training datasets including training medical images and associated non-imaging data; receiving a plurality of second feature vectors; the plurality of second feature vectors are a result of applying the first machine learning model to the plurality of second medical images; determining a similar feature vector from a plurality of second feature vectors based on the first feature vector; selecting a similar medical image from the plurality of second medical images; Similar medical images are associated with similar feature vectors, and providing a similar medical image PROV-IMG-S.

[0084] In particular, the image providing system may be configured to perform a method for providing similar medical images according to the present invention and its aspects, the image providing system being configured to perform said method and its aspects by means of its means, its interface and / or its processor configured to perform the steps of said method.

[0085] According to a fifth aspect, the present invention relates to a computer program product comprising instructions which, when said program is executed by a computer, cause said computer to carry out the method for providing a similar medical image according to the present invention and its aspects.

[0086] According to a sixth aspect, the present invention relates to a computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform a method for providing similar medical images according to the present invention and its aspects.

[0087] The implementation of the present invention or any of its aspects by a computer program product and / or a computer readable medium has the advantage that existing servers, devices and clients can be easily adapted by software updates to operate according to the proposals according to the invention.

[0088] A computer program product may be, for example, a computer program, or may contain other elements apart from the computer program, which may be hardware and / or software, for example a memory device storing the computer program or a hardware key for using the computer program, and for example documentation or a software key for using the computer program.

[0089] According to a seventh aspect, the present invention relates to a model providing system for providing a first machine learning model, the system comprising: means for receiving a training data set including training medical images and associated non-imaging data; means for determining output data by applying a second machine learning model to the training medical images; the second machine learning model includes the first machine learning model; A means for modifying parameters of the second machine learning model based on a difference between the non-imaging data and the output data; means for deriving the first machine learning model from the second machine learning model; and a means for providing a first machine learning model.

[0090] According to an eighth aspect, the present invention relates to a model providing system for providing a first machine learning model, the system comprising: receiving a training data set including training medical images and associated non-imaging data; determining output data by applying a second machine learning model to the training medical images; the second machine learning model includes the first machine learning model; modifying parameters of the second machine learning model based on differences between the non-imaging data and the output data; Deriving the first machine learning model from the second machine learning model; providing a first machine learning model, and a processor and interface configured to perform the steps of the method.

[0091] In particular, the model providing system may be configured to perform the method for providing a first machine learning model according to the present invention and its aspects, the model providing system being configured to perform the method and its aspects by means of its means, its interface and / or its processor configured to perform the steps of the method.

[0092] According to a ninth aspect, the present invention relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to perform a method for providing a first machine learning model according to the present invention and its aspects.

[0093] According to a tenth aspect, the present invention relates to a computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform a method for providing a first machine learning model according to the present invention and its aspects.

[0094] The implementation of the present invention or any of its aspects by a computer program product and / or a computer readable medium has the advantage that existing servers, devices and clients can be easily adapted by software updates to operate according to the proposals according to the invention.

[0095] A computer program product may be, for example, a computer program, or may contain other elements apart from the computer program, which may be hardware and / or software, for example a memory device storing the computer program or a hardware key for using the computer program, and for example documentation or a software key for using the computer program. [Brief description of the drawings]

[0096] The above-mentioned nature, features, and advantages of the present invention and the manner in which they are achieved will be more clearly understood and will be better understood with reference to the following detailed description and embodiments in conjunction with the drawings. The following disclosure is not intended to limit the present invention to the embodiments contained therein. In the various drawings, similar components or parts are designated by the same reference numerals. Generally, the drawings are not drawn to scale.

[0097] The numbering and / or ordering of the method steps is for ease of understanding and should not be construed to imply that the specified steps must be performed according to the numbering and / or order of their reference numbers in the figures, unless otherwise stated or implicitly apparent. In particular, some or even all of the method steps may be performed simultaneously, in an overlapping manner, or sequentially. [Figure 1] FIG. 2 is a data flow diagram according to an embodiment of the present invention. [Diagram 2] FIG. 4 is a data flow diagram according to another embodiment of the present invention. [Diagram 3] FIG. 11 is a data flow diagram according to another embodiment of the present invention relating to training a first machine learning model. [Figure 4] 1 illustrates the structures of first and second machine learning models according to an embodiment of the present invention. [Diagram 5] 4 is a flow chart of a method for providing similar medical images according to an embodiment of the present invention. [Figure 6] 4 is a flowchart of a method for providing similar medical images according to another embodiment of the present invention. [Figure 7] 1 is a flowchart of a method for providing a first machine learning model according to an embodiment of the present invention. [Figure 8] 1 shows an example of a first medical image that is a whole slide histopathology image. [Figure 9] 1 illustrates an association between a first patch of a first medical image and a second patch of a similar medical image. [Figure 10] 1 illustrates an image providing system according to an embodiment of the present invention. [Figure 11] 1 illustrates a model serving system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0098] FIG. 1 shows a data flow diagram according to one embodiment of the present invention.

[0099] In this example, the first medical image IMG-1 is used directly as an input to the first machine learning model ML-1 to generate the first feature vector FV-1. Alternatively, the first medical image IMG-1 and additional input data, such as imaging metadata and patient demographic data, can be used. In this example, the first medical image IMG-1 is a 150x150 pixel user-selected patch extracted from an H&E stained whole slide pathology image. The first feature vector FV-1 is a 512-dimensional vector with real entries.

[0100] Similarly, each second medical image IMG-2 of the plurality of second medical images IMG-2 is used as an input to the first machine learning model ML-1 to generate a plurality of second feature vectors FV-2. Alternatively, the second medical images IMG-2 and additional input data, such as imaging metadata and patient demographic data, can be used. In this embodiment, all the second medical images IMG-2 are 150x150 pixel patches extracted from the whole slide image, and all the second feature vectors FV-2 are 512-dimensional vectors with real entries.

[0101] In addition, the first feature vector FV-1 is compared with each of the second feature vectors FV-2 of the plurality of second feature vectors FV-2. In this embodiment, the sum of squared differences s(v,w i )=(vw i )·(vw i For each comparison, a similarity is calculated based on ∑ i = 1 ... i is one of the second feature vectors FV-2. From all comparisons, the one with the highest similarity (here, the smallest s(v,w i ) ≥ 0) is selected as the similar feature vector FV-S. Alternatively, instead of the sum of squared differences (SSD), a method such as that described in Section 4 of Teichmann et al., “Detect-to-Retrieve: Efficient Regional Aggregation for Image Search”, arXiv:1812.01584v2, can be used to compare the feature vector FV-1 with the set of feature vectors FV-2.

[0102] Based on the similar feature vector FV-S, the similar medical image IMG-S can be determined based on the image being used as an input to the first machine learning model ML-1 to calculate the similar feature vector FV-S. For example, a database table relating the second feature vector FV-2 and the second medical image IMG-2 can be provided as a data table that can be queried to determine the similar medical image IMG-S based on the similar feature vector FV-S.

[0103] Of course, it is also possible to select several similar feature vectors FV-S and their corresponding similar medical images IMG-S. For example, a certain number of second feature vectors FV-2 can be selected as similar feature vectors FV-S (e.g., feature vectors with the highest similarity), and all feature vectors FV-2 that meet a threshold similarity can be selected as similar feature vectors FV-S.

[0104] FIG. 2 shows a data flow diagram according to one embodiment of the present invention.

[0105] Different from the embodiment shown in FIG. 1, in this embodiment, the first medical image IMG-1 is not directly used as the input of the first machine learning model ML-1, but the first patch PTCH-1 from the first medical image IMG-1 is used. In this embodiment, the first medical image IMG-1, which is a whole slide image, is divided into first patches PTCH-1 of 150×150 pixels (which may overlap). For each of the first patches PTCH-1, a first feature vector FV-1 is determined by applying the first machine learning model ML-1. The first feature vector FV-1 is a 512-dimensional vector with real entries.

[0106] The multiple second medical images IMG-2 are also divided into patches, and a second feature vector FV-2 is calculated for each patch of each second medical image IMG-2 based on applying the first machine learning model ML-1 to the respective patch.

[0107] In this embodiment, for each first feature vector FV-1, a similar feature vector FV-S is determined from a number of second feature vectors as described with respect to Fig. 1. Alternatively, it is also possible to determine the similar feature vector FV-S for a particular subset of the first feature vectors FV-1, for example a subset where each patch corresponds to a segmented portion of a tumor in the first medical image IMG-1. Alternatively, it is also possible to determine a pair consisting of one of the first feature vectors FV-1 and one of the second feature vectors FV-2 that have the highest mutual similarity, and to select the determined second feature vector FV-2 as the similar feature vector FV-S.

[0108] At least one corresponding similar medical image IMG-S can be selected based on at least one similar feature vector FV-S. The number of similar medical images IMG-S may be less than the number of similar feature vectors FV-S if some of the similar feature vectors FV-S correspond to (different second patches PTCH-S of) the same similar medical image IMG-S. Alternatively, at least one second patch PTCH-S corresponding to the similar feature vector FV-S can be selected as the similar medical image IMG-S (i.e., only each second patch PTCH-S is extracted instead of extracting the entire similar medical image IMG-S).

[0109] FIG. 3 shows a data flow diagram according to one embodiment of the invention for training a first machine learning model ML-1.

[0110] In this embodiment, the first machine learning model ML-1 is part of the second machine learning model ML-2. In addition to the first machine learning model ML-1, the second machine learning model includes a classifier unit CU. The second machine learning model ML-2 may also include further components. In this embodiment, the second machine learning model ML-2 is a neural network including multiple layers, where the first machine learning model ML-1 includes a first subset of the successive layers, and the classifier unit CU includes a second subset of the successive layers.

[0111] In training the machine learning models ML-1 and ML-2, a training data set is used, each of which includes a training medical image IMG-T and associated non-imaging data PDAT-T. Specifically, the training medical image IMG-T and the non-imaging data PDAT-T relate to one patient. In this embodiment, the training medical image IMG-T is a 150x150 pixel user-selected patch extracted from an H&E stained whole slide pathology image. However, other types of training medical images, e.g., recorded by a computed tomography or magnetic resonance imaging device, can also be used.

[0112] The training medical image IMT-T is used as an input to the second machine learning model ML-2 and the first machine learning model ML-1. This is based on the fact that in this embodiment, the first machine learning model ML-1 is the first part of the second machine learning model ML-2 in terms of the processing direction of the feedforward network. The second machine learning model ML-2 applied to the training medical image IMT-T generates output data OUT-T. Inside, the first machine learning model ML-1 calculates a feature vector, and the classifier unit CU takes the feature vector as input and calculates the output data OUT-T as output.

[0113] In training, the output data OUT-T and the non-imaging data PDAT-T are compared, and at least one parameter of the second machine learning model ML-2 is adjusted based on the comparison. In this embodiment, the at least one parameter of the second machine learning model ML-2 is a parameter of the first machine learning model ML-1 and / or a parameter of the classifier unit CU.

[0114] In this embodiment, the non-imaging data PDAT-T includes the patient's genomic data and data related to the molecular subtype of colorectal cancer (generally speaking, any of the classes "CMS1", "CMS2", "CMS3" and "CMS4" for "hypermutation subtype", "canonical subtype", "metabolic subtype" and "mesenchymal subtype"). The genomic data refers to information on whether the patient's genome contains a mutation in the BRAF gene ("proto-oncogene B-Raf"), in particular, whether the BRAF gene contains a BRAF-V600E mutation. Specifically, the non-imaging data PDAT-T can be a three-dimensional vector p, where p1∈{0,1} according to whether a mutation exists in the patient's BRAF gene, p2∈{0,1} according to whether the patient's genome contains a BRAF-V600E mutation, and p3∈{1,2,3,4} according to the patient's molecular subtype of colorectal cancer.

[0115] In this embodiment, the output data OUT-T is also a three-dimensional vector with real numbers v1, v2 ∈ [0,1] and v3 ∈ [0,4]. During training, the parameters of the second machine learning network ML-2 are a loss function L=(p1-v1) based on the difference between the non-image data PDAT-T and the output data OUT-T. 2 +(p2-v2) 2 +(p3-v3) 2 is adjusted based on the backpropagation algorithm using

[0116] FIG. 4 illustrates a first machine learning model ML-1 and a second machine learning model ML-2 for use in one embodiment of the present invention.

[0117] In this embodiment, the second machine learning model ML-2 includes a first machine learning model ML-1 and a classifier unit CU. The first machine learning model ML-1 receives a first medical image IMG-1 as an input and generates a first feature vector FV-1 as an output, and the classifier unit CU receives the first feature vector FV-1 as an input and generates output data OUT-T corresponding to the non-imaging data PDAT-T.

[0118] In this example, the input image IMG-1 is a square patch of size 250x250 pixels selected from a stained whole slide image. The first machine learning model ML-1 includes a feature unit ML-F and an attention unit ML-A, where the attention unit ML-A is based on an attention mechanism. The feature unit ML-F takes the first medical image IMG-1 as input and generates a number of sub-features as output. The attention unit ML-A takes the sub-features as input and calculates an attention score for each of the sub-features.

[0119] In this embodiment, the feature unit ML-F is derived from the well-known ResNet50 model (e.g., as described in He et al., “Deep Residual Learning for Image Recognition”, arXiv:1512.03385). However, since the feature unit ML-F should not generate a classification value but should generate a certain number of feature vectors, a certain number of the last layers of the ResNet50 model are skipped to generate multiple sub-features. For example, 49 sub-features (f n ) may be present.

[0120] In this embodiment, the attention unit ML-A calculates an attention score function α(f n ,θ). The output of the attention unit ML-A is the weighted sum Σ n α(f n ,θ)f nand the parameter θ can be trained based on backpropagation if the output of the classifier unit CU is to be compared with the ground truth, which is the non-imaging data PDAT-T.

[0121] In this embodiment, the classifier unit CU is a fully connected network that receives 512 values ​​(corresponding to the entries of the first feature vector FV-1) as input. In other words, the input layer of the classifier unit CU has 512 nodes. The number of output values ​​corresponds to the classification items that the second machine learning model ML-2 should calculate. For example, if the non-imaging data PDAT-T is a three-dimensional vector, the output includes at least three (preferably exactly three) values, and the output layer of the classifier unit CU includes three nodes. In this case, there may be three hidden layers with 120, 40, and 15 nodes. Alternatively, the classifier unit may be a convolutional neural network.

[0122] FIG. 5 shows a flow chart of an embodiment of a method for providing similar medical images IMG-S.

[0123] The first step of the illustrated method is a step REC-IMG-1 of receiving a first medical image IMG-1 for a first patient PAT-1. In this example, the first medical image IMG-1 is a rectangular patch of 250x250 pixels selected from a whole slide digital pathology image. Alternatively, other types of first medical images IMG-1 can be used. For example, a DICOM image generated in a radiological examination using magnetic resonance imaging or computed tomography can be used. The receiving step REC-IMG-1 is performed by an interface PSYS.IF of the image providing system PSYS.

[0124] The next step of the illustrated method is a step DET-FV-1 of determining a first feature vector FV-1 by applying a first machine learning model ML-1 to a first medical image IMG-1 by a computing unit PSYS.CU of the image providing system PSYS. The machine learning model ML-1 has a structure as shown and described in FIG. 4. The first machine learning model ML-1 has been trained on training data sets, each of which includes training medical images IMG-T and associated non-imaging data PDAT-T. The training medical images IMG-T are 250×250 pixel sized patches of whole slide images, and do not specifically include medical images associated with the first patient PAT-1. Alternatively, other types of training medical images IMG-T can be used, although advantageously the size and type of the training medical images IMG-T match the size and type of the first medical image IMG-1. In this embodiment, the non-imaging data PDAT-T has a structure as described with respect to Fig. 3, alternatively, other non-imaging data PDAT-T can be used. In this embodiment, the training medical images IMG-T and the non-imaging data PDAT-T in each training data set correspond to one patient.

[0125] The next step in the embodiment shown in Fig. 5 is the step REC-FV-2 of receiving, by the interface PSYS.IF of the image providing system PSYS, a plurality of second feature vectors FV-2, which are the result of applying the first machine learning model ML-1 to a plurality of second medical images IMG-2. In this embodiment, the second medical images IMG-2 are patches of the whole slide image, of size 250x250 pixels. Alternatively, other types of second medical images IMG-2 can be used, although it is advantageous that the size and type of the second medical images IMG-2 match the size and type of the first medical image IMG-1.

[0126] In this embodiment, each of the second feature vectors FV-2 has the same structure and dimensions as the first feature vector FV-1. Specifically, each of the second feature vectors FV-2 is a 512-dimensional vector with real entries. In this embodiment, each of the second feature vectors FV-2 is a result of applying the first machine learning model ML-1 to each of the second medical images IMG-2, in other words, there is a bijective mapping between the second medical images IMG-2 and the second feature vectors FV-2.

[0127] The next step in this embodiment is a step DET-FV-S of determining a similar feature vector FV-S from a plurality of second feature vectors FV-2 based on the first feature vector FV-1 and in the computing unit PSYS.CU of the image providing system. The similar feature vector FV-S is one of the second feature vectors FV-2. Specifically, the similar feature vector FV-2 is one of the second feature vectors FV-2 that is closest to the first feature vector FV-1 according to a predefined metric. In this embodiment, Euclidean distance is used as the metric, and the similar feature vector FV-S is one of the second feature vectors FV-2 that has the smallest Euclidean distance to the first feature vector FV-1. Alternatively, other metrics such as Manhattan metric or cosine similarity can be used.

[0128] The next step of the method shown in Fig. 5 is a step SEL-IMG-S of selecting a similar medical image IMG-S from a plurality of second medical images IMG-2, the similar medical image IMG-S being associated with a similar feature vector FV-S. In this embodiment, the similar medical image IMG-S is associated with the similar feature vector FV-S by the similar feature vector FV-S calculated by applying the first machine learning model ML-1 to the similar medical image IMG-S. Specifically, the association between the second medical image IMG-2 and the second feature vector FV-2 can be stored in a database, so that the step SEL-IMG-S of selecting a similar medical image IMG-S may include accessing or querying this database to find the similar medical image IMG-S associated with the similar feature vector FV-S.

[0129] The last step in the illustrated embodiment is a step PROV-IMG-S of providing the similar medical image IMG-S at the interface PSYS.IF of the image providing system PSYS. In particular, the step PROV-IMG-S of providing the similar medical image IMG-S may include transferring the similar medical image IMG-S to another entity, storing the similar medical image IMG-S in a storage unit, and / or displaying the similar medical image IMG-S on a display unit.

[0130] 6 shows a flow chart of an embodiment of a method for providing similar medical images IMG-S, in which the steps REC-IMG-1 of receiving a first medical image IMG-1, REC-FV-2 of receiving a plurality of second feature vectors, DET-FV-S of determining similar feature vectors FV-S, SEL-IMG-S of selecting similar medical images IMG-S, and PROV-IMG-S of providing similar medical images IMG-S include advantageous features and options as described with respect to the embodiment of FIG.

[0131] In this embodiment, the method further comprises an optional step DET-SEG of determining a segmentation of the tumor in the first medical image IMG-1 by a computation unit PSYS.CU of the image providing system PSYS. The segmentation can be determined in various ways. For example, a neural network with a U-Net structure can be used that takes the first medical image IMG-1 as input and calculates a binary mask of the same size as output. In this case, pixels with a value of 1 correspond to tumor pixels and pixels with a value of 0 correspond to non-tumor pixels. Alternatively, it is also possible to use other segmentation methods that are not based on a learning algorithm.

[0132] In this embodiment, the method further includes an optional step DIV-IMG-1 of dividing the first medical image IMG-1 into a number of first patches PTCH-1 by a computing unit PSYS.CU of the image providing system PSYS. In this embodiment, a maximum number of non-overlapping first patches PTCH-1 of a predetermined pixel size (e.g., 250x250 pixels) is determined. Alternatively, a predetermined number M·N (M, N are predetermined integers) of first patches PTCH-1 can be generated by dividing the first image IMG-1 into equal parts M times along the first axis and N times along the second axis.

[0133] The optional step of determining the segmentation of the tumor DET-SEG can be performed before or after the step of segmenting the first medical image IMG-1 DIV-IMG-1, or both steps can be performed in parallel.

[0134] In this embodiment, in the step DET-FV-1 of determining a first feature vector FV-1, a first machine learning model ML-1 is applied to at least one of the first patches PTCH-1 corresponding to the segmented tumor (i.e., by overlapping with the segmented tumor mask). In particular, the first machine learning model ML-1 is applied to all of the first patches PTCH-1 corresponding to the segmented tumor, and an individual first feature vector FV-1 is calculated for all of these first patches PTCH-1. In particular, this means that a step DET-FV-1 of determining a first feature vector for each of the associated first patches PTCH-1 is performed, and a similar feature vector FV-S and a similar medical image IMG-S are determined for each of the associated first patches PTCH-S. In addition, it is also possible to determine only one similar feature vector FV-S and only one similar medical image IMG-S based on the feature vector among the multiple second feature vectors FV-2 that has the greatest similarity to any of the first feature vectors FV-1.

[0135] Further, in this embodiment, the process DET-FV-1 of determining the first feature vector FV-1 includes an optional subset DET-SF of determining a plurality of sub-features by applying a feature unit ML-F to the first medical image IMG-1, as described in relation to FIG. 4, and an optional subset DET-AM of determining the first feature vector FV-1 by applying an attention unit ML-A to the plurality of sub-features.

[0136] Furthermore, the step DET-FV-1 of determining the first feature vector FV-1 includes an optional subset RED-DIM that reduces the dimensionality of the sub-features by applying principal component analysis and / or a learnable bottleneck layer.

[0137] FIG. 7 shows a flowchart of a method for providing a first machine learning model ML-1 according to an embodiment of the present invention.

[0138] According to this embodiment, the method comprises as a first step a step T-REC of receiving a training data set comprising training medical images IMG-T and related non-imaging data PDAT-T. In particular, the training medical images IMG-T and the non-imaging data PDAT-T are related since they are obtained from one patient. The step T-REC of receiving the training data set is executed by an interface TSYS.IF of the model providing system TSYS.

[0139] The next step in the illustrated embodiment is a step T-DET of determining output data by applying a second machine learning model ML-2 to the training medical images IMG-T, the second machine learning model ML-2 including the first machine learning model ML-1. Specifically, the first machine learning model ML-1 and the second machine learning model ML-2 have the structure as described with respect to Fig. 4, so that the second machine learning model ML-2 includes the first machine learning model ML-1 and a classifier unit CU. The step T-DET of determining output data is executed by a computation unit TSYS.CU of the model providing system TSYS.

[0140] The next step in the illustrated embodiment is a step T-MDF of modifying parameters of the second machine learning model ML-2 based on the difference between the non-imaging data PDAT-T and the output data. In particular, the parameters of the second machine learning model ML-2 correspond to the weights of the edges in the neural network corresponding to the second machine learning model ML-2. In particular, the parameters of the second machine learning model ML-2 are on the other hand also the parameters of the first machine learning model ML-1. In particular, the parameters are modified based on a gradient descent method, in particular a backpropagation algorithm. The step T-MDF of modifying parameters of the second machine learning model ML-2 is executed by a computation unit TSYS.CU of the model providing system TSYS.

[0141] The next step in the illustrated embodiment is a step T-EXT of extracting the first machine learning model ML-1 from the second machine learning model ML-2. This step is executed by the computation unit TSYS.CU of the model providing system TSYS. In this embodiment, the first machine learning model ML-1 is a subpart of the second machine learning model ML-2, and the step T-EXT of extracting the first machine learning model ML-1 can correspond to removing all parts that are related to the second machine learning model ML-2 but not to the first machine learning model ML-1.

[0142] The final step in the illustrated embodiment is the step T-PROV of providing the first machine learning model ML-1, which is performed by the interface TSYS.IF of the model provision system TSYS.

[0143] The above steps correspond to one step of training the first and second machine learning models ML-1 and ML-2. To improve the performance of the first machine learning model ML-1, these steps may be performed several times on separate training datasets including the training medical images IMG-T and the associated non-imaging data PDAT-T.

[0144] In this embodiment, 1000 sets of training data sets for 1000 different patients were used. For each training data set, the non-imaging data PDAT-T is a three-dimensional vector p, where p1 ∈ {0,1} corresponds to whether a mutation exists in the patient's BRAF gene, p2 ∈ {0,1} corresponds to whether the patient's genome contains a BRAF-V600E mutation, and p3 ∈ {1,2,3,4} corresponds to the molecular subtype of the patient's colon cancer. All of these values ​​can be determined by determining the gene panel of each patient. Furthermore, for each training data set, the training medical image IMG-T is a 150×150 pixel user-selected patch extracted from the H&E stained whole slide pathology image of each patient. To increase the number of training data sets, it is also possible to select multiple patches from one whole slide pathology image.

[0145] 8 shows an example of a first medical image IMG-1, which is a two-dimensional stained whole slide image having N×N pixels. In this example, the first medical image IMG-1 is divided into 36 square first patches PTCH-1, each of which is a two-dimensional pixel image of (N / 6)×(N / 6) pixels.

[0146] The first medical image IMG-1 is based on a tissue sample obtained from a patient by biopsy. To prepare a tissue slide from the tissue sample, a section (also called a "punch biopsy" or "block") is cut out from the tissue sample with a punch tool, and the section is cut into micrometer-thick slices to form a tissue slide. Preparation of the tissue slide further includes staining the tissue slide with H&E (acronym for "hematoxylin and eosin") stain.

[0147] To generate the whole slide image first medical image IMG-1, the tissue slide stained with H&E stain was scanned with a dedicated histopathology scanner, which scans the entire tissue slide mounted on an object carrier and converts it into a color pixel image. The whole slide image first medical image IMG-1 can then be digitally processed and, in particular, archived in a suitable database.

[0148] FIG. 9 shows an example of a first medical image IMG-1 and respective similar medical images IMG-S determined by any of the methods according to the embodiments of the present invention.

[0149] In this example, a first medical image IMG-1 is composed of a number of first patches PTCH-1, and a similar medical image IMG-S is composed of a number of second patches PTCH-S, each of which is two-dimensional and contains the same number of pixels.

[0150] The similar medical image IMG-S is selected from among a plurality of second medical images IMG-2, each of which is a whole slide image of a tissue sample generated by the method described with respect to Fig. 8. The selection of the similar medical image IMG-S is based on a similarity between a first feature vector FV-1 determined based on a patch of the first medical image IMG-1 and a second feature vector FV-2 determined based on a patch of the second medical image IMG-2.

[0151] Further, in this embodiment, relationships ASSOC.1, ASSOC.2, and ASSOC.3 between patches PTCH-1.1, PTCH-1.2, and PTCH-1.3 in the first patch PTCH-1 and patches PTCH-S.1, PTCH-S.2, and PTCH-S.3 in the second patch PTCH-S are shown. In this embodiment, the relationships are shown by displaying arrows connecting patches PTCH-1.1, PTCH-1.2, and PTCH-1.3 in the first patch PTCH-1 with patches PTCH-S.1, PTCH-S.2, and PTCH-S.3 in the second patch PTCH-S.

[0152] Specifically, in this embodiment, a similar medical image IMG-S is selected from the multiple second medical images IMG-2 based on the highest similarity between a first feature vector FV-1 associated with a first patch PTCH-1.1 of the multiple first patches PTCH-1 and a similar feature vector FV-S associated with a first patch PTCH-S.1 of the multiple second patches PTCH-2. As a result, an association ASSOC.1 is displayed between the first patch PTCH-1.1 of the multiple first patches PTCH-1 and the first patch PTCH-S.1 of the multiple second patches PTCH-2.

[0153] Further, in this embodiment, additional associations ASSOC.2 and ASSOC.3 are displayed, which correspond to pairs of patches PTCH-1.2 and PTCH-1.3 among the plurality of first patches PTCH-1 and patches PTCH-S.2 and PTCH-S.3 among the plurality of second patches PTCH-2, in which the similarity between the associated feature vectors is higher than a predetermined threshold.

[0154] Fig. 10 shows an embodiment of an image providing system PSYS, and Fig. 11 shows an embodiment of a model providing system TSYS. The image providing system PSYS is configured to perform a method for providing a similar medical image IMG-S according to an embodiment of the present invention. The model providing system TSYS is configured to perform a method for providing a first machine learning model ML-1 according to an embodiment of the present invention.

[0155] The image providing system PSYS and / or the model providing system TSYS may be a (personal) computer, a workstation, a virtual machine running on a host hardware, a microcontroller, or an integrated circuit. In particular, the image providing system PSYS and / or the model providing system TSYS may be a mobile device, such as a smartphone or a tablet. Alternatively, the image providing system PSYS and / or the model providing system TSYS may be a real or virtual group of computers (the technical term for a real group of computers is a "cluster" and the technical term for a virtual group of computers is a "cloud").

[0156] Each of the image providing system PSYS and / or the model providing system TSYS may include an interface PSYS.IF, TSYS.IF, a computing unit PSYS.CU, TSYS.CU, and a memory unit PSYS.CU, TSYS.CU. The interfaces PSYS.IF, TSYS.IF may be hardware or software interfaces (PCIBus, USB, Firewire, etc.). The computing units PSYS.CU, TSYS.CU may be composed of hardware and software elements, for example a microprocessor, a CPU (acronym for "Central Processing Unit"), a GPU (acronym for "Graphical Processing Unit"), a Field Programmable Gate Array (acronym for "FPGA"), or an ASIC (acronym for "Application-Specific Integrated Circuit"). The computing units PSYS.CU, TSYS.CU may be configured to be multi-threadable, i.e. the computing units may host different computing processes simultaneously, running active and passive processes in parallel or switching between active and passive processes. The interfaces PSYS.IF, TSYS.IF, the computation units PSYS.CU, TSYS.CU and the memory units PSYS.MU, TSYS.MU may each include several sub-units arranged to perform different tasks and / or spatially separated.

[0157] The image providing system PSYS is connected to the database DB via a network NETW. Specifically, the database DB stores a plurality of second feature vectors FV-2 and / or a plurality of second medical images IMG-2. The network NETW can be implemented as a LAN (acronym for "Local Area Network"), in particular a WiFi network, or other local connection. Alternatively, the network NETW can be the Internet. In particular, the network NETW can be implemented as a VPN (acronym for "Virtual Private Network"). Alternatively, the database DB can be integrated into the image providing system PSYS, for example, the database DB can be stored in a memory unit PSYS.MU of the image providing system PSYS. In this case, the databases are connected by an internal connection.

[0158] The following specifics are also part of this disclosure. [Section 1.1] 1. A computer-implemented method for providing similar medical images IMG-S, comprising: Receiving a first medical image IMG-1 relating to a first patient PAT-1 (REC-IMG-1); determining a first feature vector FV-1 by applying a first machine learning model ML-1 to the first medical image IMG-1 (DET-FV-1); the first machine learning model ML-1 is trained based on training data sets, each of the training data sets including training medical images IMG-T and associated non-imaging data PDAT-T; receiving a plurality of second feature vectors FV-2 (REC-FV-2); the plurality of second feature vectors FV-2 are a result of applying the first machine learning model ML-1 to a plurality of second medical images IMG-2; determining a similar feature vector FV-S from the plurality of second feature vectors FV-2 based on the first feature vector FV-1 (DET-FV-S); Selecting a similar medical image IMG-S from the plurality of second medical images IMG-2 (SEL-IMG-S); the similar medical images IMG-S are associated with the similar feature vectors FV-S; Providing the similar medical image IMG-S (PROV-IMG-S). [Section 1.2] The method of claim 1.1, A method according to claim 1, wherein each second feature vector FV-2 of the plurality of second feature vectors FV-2 is a result of applying the first machine learning model ML-1 to one second medical image IMG-2 of the plurality of second medical images IMG-2. [Section 1.3] The method of claim 1.2, A method according to claim 1, wherein there is a one-to-one correspondence between said plurality of second feature vectors FV-2 and said plurality of second medical images IMG-2. [Section 1.4] The method according to any one of items 1.1 to 1.3, The non-imaging data PDAT-T has a different structure than the first feature vector FV-1. [Section 1.5] The method of claim 1.4, The non-imaging data PDAT-T has a lower dimensionality than the first feature vector FV-1. [Section 1.6] The method according to any one of items 1.1 to 1.5, The method, wherein said first medical image IMG-1 is a histopathology slide image, in particular a whole slide image. [Section 1.7] The method according to any one of items 1.1 to 1.6, The method, wherein the non-imaging data PDAT-T includes at least one of the following categories: omics data, diagnostic data, treatment history, and demographic data. [Section 1.8] The method of claim 1.7, The non-imaging data PDAT-T includes omics data, The omics data comprises at least one of genomic data, transcriptomic data, proteomic data, and metabolomic data. [Section 1.9] The method according to any one of items 1.6 to 1.8, The non-imaging data PDAT-T includes omics data, the training medical image IMG-T represents a tumor in a patient; The omics data is related to the patient's type of cancer. [Section 1.10] The method of claim 1.9, further comprising: The method, wherein said training medical images IMG-T are histopathology slide images representative of a patient's tumor cells, in particular whole slide images representative of a patient's tumor cells. [Section 1.11] The method according to any one of claims 1.1 to 1.10, the first machine learning model ML-1 is trained by training a second machine learning model ML-2 that includes the first machine learning model ML-1; The method, wherein the second machine learning model ML-2 is trained to predict non-imaging data PDAT-T based on training medical images IMG-T. [Section 1.12] The method of claim 1.11, The method, wherein the second machine learning model ML-2 is a classifier. [Section 1.13] The method of claim 1.11 or 1.12, The second machine learning model ML-2 includes the first machine learning model ML-1 and a classification unit CU; The classification unit CU processes an output of the first machine learning model ML-1 to determine a classification value. [Section 1.14] The method of claim 1.13, A method, wherein the classification unit CU comprises or consists of an additional neural network layer. [Section 1.15] The method according to any one of the above items 1.11 to 1.14, A method, wherein at least one parameter of the second machine learning model ML-2 is based on a difference between an output of the second machine learning model ML-2 when the second machine learning model ML-2 is applied to the training medical image IMG-T and the non-imaging data PDAT-T corresponding to the training medical image IMG-T. [Section 1.16] The method according to any one of claims 1.1 to 1.10, A method in which output data is generated by applying the first machine learning model ML-1 to the medical images IMG-1, IMG-2, IMG-T, the output data including the feature vectors FV-1, FV-2 and the non-imaging data PDAT-T. [Section 1.17] The method according to any one of claims 1.1 to 1.16, The first machine learning model ML-1 includes a feature unit ML-F and an attention unit ML-A; The attention unit ML-A, based on the attention mechanism, The step DET-FV-1 of determining the first feature vector FV-1 comprises: determining a plurality of sub-features by applying the feature unit ML-F to the first medical image IMG-1 (DET-SF); determining the first feature vector FV-1 by applying the attention unit ML-A to the plurality of sub-features (DET-AM). [Section 1.18] The method of claim 1.17, The step DET-FV-1 of determining the first feature vector FV-1 comprises: Reducing the dimensionality of the sub-features by applying principal component analysis and / or a learnable bottleneck layer (RED-DIM). [No. 1.19] The method of claim 1.17 or 1.18, The method, wherein the first machine learning model ML-1 is a deep local feature network. [Section 1.20] The method according to any one of claims 1.1 to 1.19, determining a segmentation of the tumor in the first medical image IMG-1 (DET-SEG); The method of claim 1, wherein in the step DET-FV-1 of determining the first feature vector FV-1, the first machine learning model ML-1 is applied to a portion of the first medical image IMG-1 that corresponds to the segmented tumor. [Section 1.21] The method according to any one of claims 1.1 to 1.19, Dividing the first medical image IMG-1 into a plurality of first patches PTCH-1 (DIV-IMG); In the step DET-FV-1 of determining the first feature vector FV-1, the first machine learning model ML-1 is applied to one of the first patches PTCH-1. [Section 1.22] The method of claim 1.21, A method according to claim 1, wherein for each of the first patches PTCH-1, a corresponding first feature vector FV-1 is determined by applying the first machine learning model ML-1 to each of the first patches PTCH-1. [Section 1.23] The method of claim 1.21, The method, wherein the first feature vector FV-1 is based on the output of the first machine learning model ML-1 when applied to each first patch in the subset of the first patches PTCH-1. [Section 1.24] The method of claim 1.23, The subset of the first patches PTCH-1 is selected based on an attention mechanism of the first machine learning model ML-1. [Section 1.25] The method of claim 1.21 or 1.22, comprising The similar medical image IMG-S includes a plurality of second patches PTCH-S; one of the second patches PTCH-S is associated with the similar feature vector FV-S; The method, wherein the step of providing the similar medical image IMG-S PROV-IMG-S includes displaying an association between one of the first patches PTCH-1 and one of the second patches PTCH-S. [Section 1.26] The method of claim 1.25, The method, wherein the step of providing the similar medical image IMG-S PROV-IMG-S includes displaying an association between the first patch PTCH-1 and the second patch PTCH-S based on a similarity of the feature vectors FV-1, FV-S associated with the first patch PTCH-1 and the second patch PTCH-S. [Section 2.1] 1. A computer-implemented method for providing a first machine learning model, comprising: receiving a training dataset including training medical images IMG-T and associated non-imaging data PDAT-T (T-REC); Determining output data by applying a second machine learning model ML-2 to the training medical images IMG-T (T-DET); the second machine learning model ML-2 includes the first machine learning model ML-1; Modifying parameters of the second machine learning model ML-2 based on a difference between the non-imaging data PDAT-T and the output data (T-MDF); Extracting the first machine learning model ML-1 from the second machine learning model ML-2 (T-EXT); Providing the first machine learning model ML-1 (T-PROV). [Section 2.2] The method of claim 2.1, The method of claim 1, wherein the first machine learning model ML-1 includes parameters of the second machine learning model ML-2. [Section 2.3] The method of claim 2.1 or 2.2, The method of claim 1, wherein the step of modifying parameters of the second machine learning model ML-2 (T-MDF) is based on gradient descent, in particular backpropagation. [Section 3.1] An image providing system PSYS that provides a similar medical image IMG-S, means for receiving (REC) a first medical image IMG-1 relating to a first patient PAT-1; means for determining a first feature vector FV-1 (DET-FV-1) by applying a first machine learning model ML-1 to the first medical image IMG-1; the first machine learning model ML-1 is trained based on training data sets, each of the training data sets including training medical images IMG-T and associated non-imaging data PDAT-T; means for receiving a plurality of second feature vectors FV-2 (REC-FV-2); the plurality of second feature vectors FV-2 are a result of applying the first machine learning model ML-1 to a plurality of second medical images IMG-2; a means for determining a similar feature vector FV-S from the plurality of second feature vectors FV-2 based on the first feature vector FV-1 (DET-FV-S); a means for selecting (SEL-IMG-S) a similar medical image IMG-S from the plurality of second medical images IMG-2; the similar medical images IMG-S are associated with the similar feature vectors FV-S; and a means for providing the similar medical image IMG-S (PROV-IMG-S). [Section 3.2] An image providing system PSYS that provides a similar medical image IMG-S, Receiving (REC) a first medical image IMG-1 relating to a first patient PAT-1; determining a first feature vector FV-1 by applying a first machine learning model ML-1 to the first medical image IMG-1 (DET-FV-1); the first machine learning model ML-1 is trained based on training data sets, each of the training data sets including training medical images IMG-T and associated non-imaging data PDAT-T; receiving a plurality of second feature vectors FV-2 (REC-FV-2); the plurality of second feature vectors FV-2 are a result of applying the first machine learning model ML-1 to a plurality of second medical images IMG-2; determining a similar feature vector FV-S from the plurality of second feature vectors FV-2 based on the first feature vector FV-1 (DET-FV-S); Selecting a similar medical image IMG-S from the plurality of second medical images IMG-2 (SEL-IMG-S); the similar medical images IMG-S are associated with the similar feature vectors FV-S; providing the similar medical image IMG-S (PROV-IMG-S); and an interface PSYS.IF. [Section 3.3] The image providing system PSYS according to the above item 3.1 or 3.2, An image providing system PSYS comprising additional means for carrying out the method according to any one of paragraphs 1.1 to 1.26 or configured to carry out the method according to any one of paragraphs 1.1 to 1.26. [Section 3.4] A computer program product (computer program) including instructions, A computer program product, the instructions which, when executed by a computer, cause the computer to perform the method of any one of clauses 1.1 to 1.26. [Section 3.5] A computer-readable medium containing instructions, A computer-readable recording medium, the instructions, when executed by a computer, causing the computer to perform the method of any one of clauses 1.1 to 1.26. [Section 4.1] A model provision system TSYS that provides a first machine learning model ML-1, means for receiving a training data set (T-REC) including training medical images IMG-T and associated non-imaging data PDAT-T; A means for determining output data (T-DET) by applying a second machine learning model ML-1 to the training medical image IMG-T; The second machine learning model ML-2 includes the first machine learning model ML-1; A means for modifying parameters of the second machine learning model ML-2 based on a difference between the non-imaging data PDAT-T and the output data (T-MDF); A means for extracting the first machine learning model ML-1 from the second machine learning model ML-2 (T-EXT); A model providing system TSYS including: a means for providing the first machine learning model ML-1 (T-PROV). [Section 4.2] A model provision system TSYS that provides a first machine learning model ML-1, receiving a training dataset including training medical images IMG-T and associated non-imaging data PDAT-T (T-REC); Determining output data by applying a second machine learning model ML-1 to the training medical images IMG-T (T-DET); The second machine learning model ML-2 includes the first machine learning model ML-1; Modifying parameters of the second machine learning model ML-2 based on a difference between the non-imaging data PDAT-T and the output data (T-MDF); Extracting the first machine learning model ML-1 from the second machine learning model ML-2 (T-EXT); Providing the first machine learning model ML-1 (T-PROV); and an interface TSYS.IF. [Section 4.3] A model providing system TSYS according to claim 4.1 or 4.2, A system for providing a system TSYS, comprising additional means for carrying out the method according to any one of the preceding paragraphs 2.1 to 2.3 or configured to carry out the method according to any one of the preceding paragraphs 2.1 to 2.3. [Section 4.4] A computer program product (computer program) including instructions, A computer program product, the instructions which, when executed by a computer, cause the computer to perform the method of any one of clauses 2.1 to 2.3. [Section 4.5] A computer-readable medium containing instructions, A computer-readable recording medium, the instructions, when executed by a computer, causing the computer to perform the method of any one of paragraphs 2.1 to 2.3.

[0159] Unless expressly stated otherwise, the individual embodiments, or individual aspects and features thereof, can be combined or substituted with one another without limiting or expanding the scope of the invention disclosed, and such combinations or substitutions are meaningful within the spirit of the invention. Advantages described with respect to one embodiment of the invention are also advantages of other embodiments of the invention, to the extent applicable.

Claims

1. 1. A computer-implemented method for providing similar medical images (IMG-S), comprising: receiving (REC-IMG-1) a first medical image (IMG-1) relating to a first patient (PAT-1); determining a first feature vector (FV-1) by applying a first machine learning model (ML-1) to the first medical image (IMG-1) (DET-FV-1); the first machine learning model (ML-1) is trained based on training datasets, each of the training datasets including training medical images (IMG-T) and associated non-imaging data (PDAT-T); receiving a plurality of second feature vectors (FV-2) (REC-FV-2); the plurality of second feature vectors (FV-2) are a result of applying the first machine learning model (ML-1) to a plurality of second medical images (IMG-2); determining a similar feature vector (FV-S) from the plurality of second feature vectors (FV-2) based on the first feature vector (FV-1) (DET-FV-S); selecting a similar medical image (IMG-S) from the plurality of second medical images (IMG-2) (SEL-IMG-S); the similar medical images (IMG-S) are associated with the similar feature vectors (FV-S); providing said similar medical image (IMG-S) (PROV-IMG-S).

2. The method of claim 1, wherein said first medical image (IMG-1) is a histopathology slide image, in particular a whole slide image.

3. The method of claim 1 or 2, wherein the non-imaging data (PDAT-T) includes at least one of the following categories: omics data, diagnostic data, medical history, and demographic data.

4. The method of claim 3, wherein the non-imaging data (PDAT-T) comprises omics data, the omics data comprising at least one of genomic data, transcriptomic data, proteomic data, and metabolomic data.

5. The first machine learning model (ML-1) is trained by training a second machine learning model (ML-2) that includes the first machine learning model (ML-1); The method according to any one of claims 1 to 4, wherein the second machine learning model (ML-2) is trained to predict non-imaging data based on medical images, the second machine learning model (ML-2) being preferably a classifier.

6. The first machine learning model (ML-1) includes a feature unit (ML-F) and an attention unit (ML-A); The attention unit (ML-A) is based on an attention mechanism, The step (DET-FV-1) of determining the first feature vector (FV-1) comprises: determining a plurality of sub-features (DET-SF) by applying the feature unit (ML-F) to the first medical image (IMG-1); The method according to any one of claims 1 to 5, comprising determining the first feature vector (FV-1) by applying the attention unit (ML-A) to the plurality of sub-features (DET-AM).

7. The step (DET-FV-1) of determining the first feature vector (FV-1) comprises: The method of claim 6, comprising reducing the dimensionality of the sub-features by applying principal component analysis and / or a learnable bottleneck layer (RED-DIM).

8. The method according to claim 6 or 7, wherein the first machine learning model (ML-1) is a deep local feature network.

9. determining a segmentation of the tumor in said first medical image (IMG-1) (DET-SEG); 9. The method according to claim 1, wherein in the step (DET-FV-1) of determining the first feature vector (FV-1), the first machine learning model (ML-1) is applied to a portion of the first medical image (IMG-1) that corresponds to the segmented tumor.

10. Dividing (DIV-IMG-1) the first medical image (IMG-1) into a plurality of first patches (PTCH-1), The method according to any one of claims 1 to 8, wherein in the step (DET-FV-1) of determining the first feature vector (FV-1), the first machine learning model (ML-1) is applied to one of the first patches (PTCH-1).

11. the similar medical image (IMG-S) includes a plurality of second patches (PTCH-S), one of the second patches (PTCH-S) being associated with the similar feature vector (FV-S); 11. The method of claim 10, wherein the step of providing similar medical images (IMG-S) (PROV-IMG-S) includes displaying an association between one of the first patches (PTCH-1) and one of the second patches (PTCH-S).

12. 1. A computer-implemented method for providing a first machine learning model (ML-1), comprising: receiving a training data set (T-REC) including training medical images (IMG-T) and associated non-imaging data (PDAT-T); determining output data (OUT-T) by applying a second machine learning model (ML-2) to the training medical images (IMG-T); The second machine learning model (ML-2) includes the first machine learning model (ML-1); Modifying parameters of the second machine learning model (ML-2) based on a difference between the non-imaging data (PDAT-T) and the output data (OUT-T) (T-MDF); Extracting the first machine learning model (ML-1) from the second machine learning model (ML-2) (T-EXT); providing the first machine learning model (ML-1) (T-PROV).

13. An image providing system (PSYS) for providing similar medical images (IMG-S), means for receiving (REC) a first medical image (IMG-1) relating to a first patient (PAT-1); means for determining (DET-FV-1) a first feature vector (FV-1) by applying a first machine learning model (ML-1) to the first medical image (IMG-1); the first machine learning model (ML-1) is trained based on training datasets, each of the training datasets including training medical images (IMG-T) and associated non-imaging data (PDAT-T); means for receiving (REC-FV-2) a plurality of second feature vectors (FV-2); the plurality of second feature vectors (FV-2) are a result of applying the first machine learning model (ML-1) to a plurality of second medical images (IMG-2); a means for determining (DET-FV-S) a similar feature vector (FV-S) from the plurality of second feature vectors (FV-2) based on the first feature vector (FV-1); a means for selecting (SEL-IMG-S) a similar medical image (IMG-S) from the plurality of second medical images (IMG-2); the similar medical images (IMG-S) are associated with the similar feature vectors (FV-S); and a means for providing (PROV-IMG-S) said similar medical image (IMG-S).

14. A computer program comprising instructions, A computer program product, the instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 11.

15. A computer-readable medium containing instructions, A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 11.

Images