Providing similar medical images

Abstract

The invention relates to a method and system for providing similar medical images. Methods and systems include receiving a first medical image associated with a first patient. The method and system also include determining a first feature vector by applying a first machine learning model to the first medical image. The first machine learning model is trained based on training datasets, each training dataset including training medical images and associated non-imaging data. The method and system also include receiving a plurality of second feature vectors. The plurality of second feature vectors are results obtained by applying the first machine learning model to the plurality of second medical images. The method and system also include determining a similar feature vector from the plurality of second feature vectors based on the first feature vector. The method and system also include selecting a similar medical image from the plurality of second medical images, where the similar medical image is related to a similar feature vector. The methods and systems also include providing similar medical images.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging. Background Art

[0002] Independent of grammatical term usage, individuals with masculine or feminine identities are also included in the term.

[0003] In particular, in the context of cancer diagnosis and treatment planning, finding similar patients is an important topic in medical technology, especially when faced with rare or unusual cases. For example, identifying patients with similar molecular composition and tumor characteristics is crucial in guiding medical professionals in choosing effective treatment options and enhancing clinical decision making. In such complex cases, standard protocols may not be sufficient, and medical professionals may struggle with uncertainty about the best course of action. In particular, it is desirable to be able to find similar patients based on comparing corresponding medical images (e.g., histopathology slide images).

[0004] Historically, pathologists have relied primarily on manual examination of histopathology slides under a microscope to identify and compare patterns and features within tumor regions. Typically, findings are discussed at tumor board meetings that include medical experts from various fields, where additional data and modalities are considered. Tumor boards can also be used to identify similar patients. However, this approach is labor-intensive, time-consuming, and can be prone to inter- and intra-observer variability, which can lead to inconsistencies in patient evaluation and treatment decisions.

[0005] Content-based image retrieval systems have been employed to find similar images in medical databases using global features such as color, texture, and shape. While these systems have proven useful in some cases, they may not effectively match patients based on local patterns and features within the tumor region, and they lack data integration with other factors to guide the retrieval process. In digital pathology, these systems rely primarily on visual features and do not incorporate other data, which can result in retrieved images that are visually similar to each other but represent diagnostically different cancers. Summary of the invention

[0006] The problem of the present invention is to improve the finding of similar patients based on medical images.

[0007] The problem is solved according to an embodiment of the invention. Further advantageous embodiments and additional advantageous features are described in the description.

[0008] In the following, the solution according to the invention will be described for the claimed system and for the claimed method. Features, advantages or alternative embodiments herein may be assigned to other corresponding claimed objects, and other corresponding claimed objects may be assigned to features, advantages or alternative embodiments herein. In other words, the system may be improved by features described or claimed in the context of the corresponding method. In this case, the functional features of the method are embodied by the target unit of the system.

[0009] In addition, in the following, the solution according to the present invention is described for a method and system for providing similar medical images and for a method and system for providing a trained machine learning model. The features, advantages or alternative embodiments herein may be assigned to other claimed objects, and other claimed objects may be assigned to the features, advantages or alternative embodiments herein. In other words, the technical solutions and implementations for providing a trained machine learning model may be improved by features described or claimed in the context of providing similar medical images. In particular, the data set used in the method and system for providing similar medical images may have the same properties and characteristics as the corresponding data set used in the method and system for providing a trained machine learning model, and the trained machine learning model provided by the corresponding method and system may be used in the method or system for providing similar medical images.

[0010] In the following, the term "particularly" is used to indicate optional and / or advantageous additional features. In addition, the term "applying a model to data" or "applying a unit to data" is used to indicate that the corresponding data is used as input data for a model or unit, or to indicate the use of input data including the corresponding data (and potentially other data).

[0011] In a first aspect, the present invention relates to a computer-implemented method for providing similar medical images. The method comprises receiving a first medical image associated with a first patient. The method further comprises determining a first feature vector by applying a first machine learning model to the first medical image. The first machine learning model is trained based on a training data set, each training data set comprising a training medical image and associated non-imaging data. The method further comprises receiving a plurality of second feature vectors. The plurality of second feature vectors are the result of applying the first machine learning model to a plurality of second medical images. The method further comprises determining a similar feature vector from the plurality of second feature vectors based on the first feature vector. The method further comprises selecting similar medical images from the plurality of second medical images, wherein the similar medical images are associated with similar feature vectors. The method further comprises providing similar medical images. In particular, providing similar medical images may comprise storing, transmitting and / or displaying similar medical images.

[0012] In particular, the step of receiving the first medical image and the step of receiving the plurality of second feature vectors are performed by an input interface, in particular by an input interface of an image providing system. In particular, the steps of determining the first feature vector, determining the similar feature vector based on the first feature vector, and selecting the similar medical image are performed by a computing unit, in particular by a computing unit of the image providing system. In particular, the step of providing the similar medical image is performed by an output interface, in particular by an output interface of the image providing system.

[0013] In particular, the medical image is an image of a patient recorded by a medical imaging modality. The medical image may be an X-ray image, a computed tomography image (abbreviated as "CT image"), a magnetic resonance image (abbreviated as "MR image"), a positron emission tomography image (abbreviated as "PET image"), a single photon emission computed tomography (abbreviated as "SPECT image") and / or an ultrasound image (abbreviated as "US image"). Furthermore, the medical image may be a microscopic image, a histopathological image and / or a time-continuous biosignal analysis image. In particular, the medical image may be a full-slice image.

[0014] In particular, the medical image may be a two-dimensional image, a three-dimensional image or a four-dimensional image. In particular, in a four-dimensional image, there are three spatial dimensions and one time dimension.

[0015] Medical images may include a plurality of pixels or voxels, wherein the term "pixel" is primarily used for building blocks of two-dimensional images and the term "voxel" is used for building blocks of images with arbitrary dimensions (primarily for three or more dimensions). However, in the following, the term "pixel" will be used as a synonym for the term "pixel and / or voxel". Each pixel may include at least one intensity value corresponding to a certain tissue property (e.g., Hounsfield units for CT images).

[0016] A medical image may be identical to or encapsulated in one or more DICOM files. Whenever DICOM is mentioned herein, it is understood that this refers to the "Digital Imaging and Communications in Medicine" (DICOM) standard, for example according to the current DICOM PS3.1 2020c standard (or any later or earlier version of said standard). Several medical images may also be encapsulated in a single DICOM file.

[0017] In particular, the medical image may include additional metadata, such as a patient identifier, the patient's name, an identifier of the modality used to acquire the medical image, the date and time of image acquisition, the imaging protocol used during acquisition of the medical image, and / or other metadata related to the medical image and / or its acquisition process.

[0018] In particular, the first medical image, the second medical image, the similar medical image and the training medical image are medical images of the same type, i.e., the first medical image, the second medical image, the similar medical image and the training medical image are recorded with the same type of imaging modality (e.g., all medical images are recorded by CT, or all medical images are full-slice images). In particular, the first medical image, the second medical image, the similar medical image and the training medical image may have the same dimension. In particular, the first medical image, the second medical image, the similar medical image and the training medical image may have the same size (e.g., measured in the number of pixels) along each of their dimensions. In particular, the second medical image set may include the training medical image set, and / or the second medical image set may be the same as the training medical image set.

[0019] Typically, machine learning models mimic the cognitive functions that humans associate with other human minds. In particular, by being trained based on training data, machine learning models are able to adapt to new environments and detect and infer patterns. Another term for "machine learning model" is "training function."

[0020] Typically, the parameters of a machine learning model can be adjusted by means of training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and / or active learning can be used. In addition, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of a machine learning model can be iteratively adjusted through several training steps. In particular, within the training, a certain cost function can be minimized. In particular, within the training of a neural network, a back-propagation algorithm can be used.

[0021] In particular, the machine learning model may include a neural network, a support vector machine, a decision tree and / or a Bayesian network, and / or the machine learning model may be based on k-means clustering, Q-learning, a genetic algorithm and / or association rules. In particular, the neural network may be a deep neural network, a convolutional neural network or a convolutional deep neural network. In addition, the neural network may be an adversarial network, a deep adversarial network and / or a generative adversarial network.

[0022] In the context of the present disclosure, the phrase "applying a machine learning model to a medical image" should be understood as there may be additional data that can be used together with the medical image as an input to the machine learning model. Furthermore, in the context of the present disclosure, the phrase "applying a machine learning model to a medical image" should be understood as at least a portion of the medical image (possibly together with the additional data) being used as an input to the machine model. In other words, the phrase "applying a machine learning model to a medical image" should be interpreted as "applying the machine learning model to input data, wherein the input data comprises at least a portion of the medical image".

[0023] Typically, features are individual measurable attributes or characteristics of a phenomenon. In the context of the present disclosure, features may be individual measurable attributes or characteristics associated with a medical image. In particular, feature vectors are n-dimensional vectors representing numerical features of the same object (the vector space associated with these vectors may be denoted as a "feature space"). In the context of the present disclosure, feature vectors may be n-dimensional vectors representing numerical features of a medical image. In the context of the present disclosure, each ordered set of numerical values ​​may be interpreted as a feature vector.

[0024] In particular, the first eigenvector, the similar eigenvector and the second eigenvector may have the same dimension. In other words, the first eigenvector, the similar eigenvector and the second eigenvector may be elements of the same eigenspace.

[0025] In particular, the non-imaging data is medical data. In particular, the non-imaging data is medical data related to a patient. In particular, the non-imaging data is medical data that does not include a medical image. However, the non-imaging data can be derived from a medical image (e.g., a diagnosis derived by a radiologist or pathologist based on a medical image). In particular, the non-imaging data can include at least one numerical value and / or can be mapped to at least one numerical value. In particular, the non-imaging data can be an m-dimensional vector, each value of which is related to a specific patient.

[0026] The inventors have recognized that by searching for similar images or similar patients using feature vectors created by a first machine learning model trained on pairs of imaging and non-imaging data, more relevant similar images can be determined. This is due to the fact that the non-imaging data can be incorporated into the parameters of the first machine learning model within training, so that the feature vectors extracted by the first machine learning model are configured (through the training process) to be predictive not only of the visual impression of the medical images but also of the corresponding non-imaging data of the same patient.

[0027] In particular, if the proposed method for finding similar images or similar patients associated with a certain disease (e.g., a certain type of cancer) is used, based on the proposed training, the first machine learning model can skillfully identify diagnostically relevant morphological features and patterns, and accurately retrieve patients with diseases that show similarities in both morphological and non-imaging components. This advancement provides significant clinical benefits and usefulness by enabling healthcare professionals to determine personalized treatment options, improve diagnostic accuracy, and ultimately improve patient outcomes, particularly in rare and complex diseases.

[0028] According to another possible aspect of the present invention, each of the plurality of second feature vectors is a result of applying the first machine learning model to exactly one of the plurality of second medical images. In particular, there may be a one-to-one correspondence between the plurality of second feature vectors and the plurality of second medical images.

[0029] According to another possible aspect of the invention, the non-imaging data has a different structure than the first eigenvector. In particular, the non-imaging data has a lower dimension than the first eigenvector. In particular, the non-imaging data has a lower dimension than the first eigenvector in the case where both the non-imaging data and the first eigenvector can be interpreted as numerical vectors and the dimension of the vector representing the non-imaging data is smaller than the dimension of the eigenvector or the vector representing the eigenvector.

[0030] According to another aspect of the present invention, the first medical image is a histopathology slice image, in particular a whole slice image.

[0031] The histopathology slice image can be, in particular, a two-dimensional pixel image. The histopathology slice image can map at least a portion of a tissue slice prepared from a patient's tissue sample. Preparing tissue slices from tissue samples can include (e.g., with a punching tool) cutting out parts (also referred to as "punch biopsy" or "blocks") from tissue samples, wherein the parts are cut into micron-thick slices (tissue slices). The preparation of tissue slices can also include staining the tissue slices with histopathology staining. In this case, staining can be used to highlight different structures in the tissue slices such as, for example, cell walls or nuclei, or to check medical indications such as, for example, cell proliferation levels. In this process, different histopathology stains can be used for different tissues.

[0032] For the creation of one or more histopathology slice images, the tissue slice dyed with the first histopathology staining can or may have been digitized or scanned.For this reason, with suitable digitizing station such as for example referred to as full-slice scanner, the first tissue slice is imaged, which preferably scans the whole tissue slice installed on the object carrier, and converts it into pixel image.In order to obtain the staining effect by histopathology staining, the pixel image is preferably a color pixel image.Due to the overall impression of tissue and the finely resolved cellular structure both being important in diagnosis, histopathology slice images have a very high pixel resolution usually.Histopathology slice images can be digitally processed subsequently, and histopathology slice images are archived in a suitable database in particular.Under the microscope, histopathology slice images show the fine tissue structure of tissue samples, and particularly the cells included in cellular structure or tissue samples.On a larger length scale, histopathology slice images show superior tissue structure and feature, such as for example continuous regions such as for example tumor region or stroma region on density and cell morphology.

[0033] In particular, the first medical image is a histopathology section image depicting a patient tissue section stained with a histopathology stain. In particular, the histopathology stain may be H&E (abbreviation for "hematoxylin and eosin") staining or IHC (abbreviation for "immunohistochemistry") staining.

[0034] In H&E staining, hematoxylin stains the cell nucleus, and eosin stains the extracellular matrix and cytoplasm. H&E staining is the most widely used stain in digital pathology, which makes the method that provides similar images widely applicable.

[0035] IHC staining involves the process of selectively identifying antigens (proteins) in cells of a tissue section by utilizing the principle that antibodies specifically bind to antigens in biological tissues. Through this, structures that cannot be accessed with other stains such as H&E and that may be highly relevant in the context of a specific disease and should therefore be retrieved for a certain similar patient can be highlighted.

[0036] In particular, IHC staining can include a biomarker (for example, in the form of an antibody) configured to target (cytoskeleton) keratin. Keratin forms a part for cytoskeleton, and limits the mechanical properties of cells. Therefore, the abundance of keratin constitutes a good tumor marker, because keratin expression levels often change in tumor cells. Therefore, using keratin IHC staining can make it possible to identify adenocarcinoma (tumor formation of epithelial tissue). Specifically, IHC staining can include keratin biomarkers for different keratin forms, such as CK-5, CK-8, CK-14, CK-18 (wherein, " CK " represents " cytoskeleton keratin "). In itself, IHC staining can include different or additional biomarkers, such as p63 and AMACR biomarkers.

[0037] A whole slide image is a histopathology image that maps a whole tissue slide prepared from a tissue sample of a patient. A whole slide image may have a size of at least 4.000 x 4.000 pixels, or at least 10.000 x 10.000 pixels, or at least 1E6 x 1E6 pixels.

[0038] In particular, when the first medical image is a histopathological slice image, each training medical image, each of the plurality of second medical images, and / or similar medical images are also histopathological slice images. In particular, when the first medical image is a full slice image, each training medical image, each of the plurality of second medical images, and / or similar medical images are also full slice images.

[0039] The inventors recognize that histopathology slice images or full slice images can depict detailed morphological structures or other non-patient data of cells that are subject to certain diseases. Based on those morphological structures, very relevant similar medical images can be provided based on certain morphological structures or other non-patient data indicating certain diseases.

[0040] According to another aspect of the invention, the non-imaging data comprises at least one of the following categories: omics data, diagnostic data, treatment history, and / or demographic data.

[0041] In general, the term "omics data" refers to analysis results related to a patient's biomolecules, for example, analysis results generated by various "omics" technologies such as genomics, transcriptomics, proteomics and metabolomics. Biomolecules include genes, RNA transcripts, proteins and / or metabolites. Omics data can provide a comprehensive understanding of functions and interactions within biological systems. The inventors have recognized that omics data can reveal certain details about a patient's biological expression that can be advantageously used to determine feature vectors suitable for finding similar medical images. In particular, omics data describes the functional composition of diseases such as cancer, so that search terms with similar omics structures have specific diagnostic value.

[0042] In general, the term "diagnostic data" refers to data related to the identification and classification of a patient's disease, disorder, and / or physical condition. Diagnostic data includes various types of data, including medical records, diagnostic test results, patient symptoms, and clinical observations. The inventors have recognized that diagnostic data gives detailed and accurate information about a patient and / or a certain disease of the patient, so that the diagnostic data can be advantageously used to determine feature vectors suitable for finding similar medical images.

[0043] In general, the term "treatment history" refers to a record of at least a subset of the medical interventions and treatments that a patient has undergone for a particular disease, condition, or ailment. Treatment history data may include information such as the type of treatment administered, dosage, duration, treatment results, and any adverse reactions or side effects experienced. The inventors have recognized that treatment history yields valuable information about a patient's disease and its response to certain treatment options. Treatment data can be advantageously used to determine feature vectors suitable for finding similar medical images, and also give hints of successful and unsuccessful treatments for similar patients.

[0044] In general, the term "demographic data" refers to a set of characteristics and information about a patient that provides insight into their personal background, such as social, cultural, and economic background. Demographic data may include factors such as age, gender, race, and / or ethnicity, and may also include geographic location, education level, income, and occupation. The inventors have recognized that demographic data may indicate the similarity of certain patients and may be advantageously used to determine feature vectors suitable for finding similar medical images.

[0045] According to another aspect of the present invention, the non-imaging data includes omics data, and the omics data includes at least one of genomic data, transcriptomic data, proteomic data, and metabolomic data.

[0046] In general, the term "genomic data" refers to at least a portion of a patient's set of genetic information or DNA (abbreviation for deoxyribonucleic acid) sequence. Genomic data may include at least a portion of a patient's genome, including the arrangement and composition of genes, variations and mutations within the DNA. The inventors recognize that genomic data indicates a patient's genetic traits, susceptibility to disease, and response to drugs, and can therefore be advantageously used to determine feature vectors suitable for finding similar medical images.

[0047] In general, the term "transcriptomic data" refers to a set of information about the expression of genes in a particular tissue or cell type of a patient. Transcriptomic data can include the results of an analysis of RNA molecules transcribed from genes that know which genes are active and produce proteins in a given biological sample of a patient. The inventors recognized that transcriptomic data indicates which genetic information of the data is actually expressed in a particular patient, and can therefore be advantageously used to determine feature vectors suitable for finding similar medical images.

[0048] In general, the term "proteomic data" refers to a set of information about proteins present in a particular tissue, cell or biological sample of a patient. Proteomic data may include the type, quantity, modification and interaction of proteins within a patient's biological sample. The inventors recognized that proteomic data provides valuable insights into functional aspects of proteins, roles of proteins in cellular processes, and involvement of proteins in diseases, and therefore can be advantageously used to determine feature vectors suitable for finding similar medical images.

[0049] Typically, the term "metabolomic data" refers to information related to molecules (known as metabolites) present in a patient's biological sample, such as blood, urine, or tissue. Metabolomic data may include identification and quantification of metabolites and / or the patient's overall metabolic profile. The inventors recognize that metabolomic data provides valuable information about metabolic states, disease markers, and responses to treatments associated with the patient, and therefore can be advantageously used to determine feature vectors suitable for finding similar medical images.

[0050] According to another possible aspect of the invention, the non-imaging data comprises omics data, and the training medical image shows a tumor of the patient, and wherein the omics data relates to a type of cancer of the patient.

[0051] The inventors have realised that omics data may give detailed information about specific properties of a patient's cancer, which may be used to determine feature vectors suitable for finding similar medical images associated with the same patient type.

[0052] According to another possible aspect of the present invention, the training medical image is a histopathological slice image showing tumor cells of a patient, in particular a whole slice image showing tumor cells of a patient.

[0053] The inventors have realised that an image of a histopathological section showing a patient's tumour comprises detailed information on the morphology of the tumour cells which can be used to determine a feature vector suitable for finding similar medical images.

[0054] According to another aspect of the invention, the first machine learning model is trained by training a second machine learning model comprising the first machine learning model, wherein the second machine learning model is trained to predict non-imaging data based on medical images, wherein, advantageously, the second machine learning model is a classifier.

[0055] According to another possible aspect of the present invention, the second machine learning model comprises the first machine learning model and a classification unit, wherein the classification unit processes the output of the first machine learning model and determines the classification value. In particular, the classification unit comprises or consists of an additional neural network layer.

[0056] According to another possible aspect of the invention, at least one parameter of the second machine learning network is based on a difference between an output of the second machine learning network when the second machine learning network is applied to a training medical image and non-imaging data corresponding to the training medical image.

[0057] According to another aspect of the present invention, by applying a first machine learning network to a medical image, output data is generated, the output data comprising a feature vector and non-imaging data. In particular, the feature vector and the non-imaging data are different.

[0058] The inventors have realised that by using a first machine learning network that outputs both non-imaging data and feature vectors, the first machine learning network can be used to both determine similar images as well as determine corresponding non-imaging data.

[0059] According to another aspect of the present invention, the first machine learning model includes a feature unit and an attention unit, wherein the attention unit is based on an attention mechanism. According to this aspect, the step of determining a first feature vector includes: determining a plurality of sub-features by applying the feature unit to the first medical image; and determining the first feature vector by applying the attention unit to the plurality of sub-features.

[0060] In particular, each of the sub-features may be a vector having the same dimension as the first feature vector.

[0061] Typically, an attention mechanism is a subunit of a machine learning model that allows the machine learning model to focus on specific parts of the input to the attention mechanism by assigning different weights to different parts of the input to the attention mechanism. This weighting is typically based on the relevance of each part of the input to the task to be performed by the machine learning model. In a neural network, an attention mechanism may correspond to at least one network layer. In particular, an attention mechanism may refer to self-attention, dot product attention, scaled dot product attention, multi-head attention, and / or structured attention.

[0062] The inventors have recognized that by using an attention mechanism within a machine learning model, the accuracy of the predictions of the machine learning model is improved by allowing the machine learning model to focus on the most relevant information. In addition, the attention mechanism can make the machine learning model more efficient by processing only the most important data to reduce the required computing resources and make the machine learning model more scalable. In addition, the attention weights learned by the model can provide insights into which parts of the data are the most important, which can help improve the interpretability of the machine learning model.

[0063] According to another aspect of the present invention, the step of determining the first feature vector comprises reducing the dimensionality of the sub-features by principal component analysis and / or applying a learnable bottleneck layer.

[0064] Typically, principal component analysis is an orthogonal linear transformation that transforms input data into a new coordinate system such that the maximum variance of some scalar projections of the data is mapped to a first coordinate (called the first principal component), the second maximum variance is mapped to a second coordinate, etc. Based on principal component analysis, dimensionality can be reduced by retaining only a certain number of first coordinates. Typically, a learnable bottleneck layer is a layer of a neural network that includes fewer output nodes than input nodes.

[0065] The inventors have realised that by using dimensionality reduction techniques to store feature vectors (particularly those associated with a plurality of second medical images) less memory capacity is required and less computing power is required to compare feature vectors. This allows similar medical images to be provided faster and / or with greater accuracy.

[0066] According to another aspect of the present invention, the first machine learning model is a deep local feature network. The deep local feature network and its benefits are described in the article "Large-Scale Image Retrieval with Attentive Deep Local Features" arXiv:1612.06321 (2016) by H. Noh et al. This article is incorporated into the present disclosure by reference.

[0067] According to another aspect of the invention, the method comprises determining a segmentation of a tumor within a first medical image, wherein, in the step of determining a first feature vector, a machine learning model is applied to a portion of the first medical image corresponding to the segmented tumor.

[0068] In particular, the portion of the first medical image corresponding to the tumor may be a rectangular or square tile of the image. In particular, the rectangular or square tile may contain all segmented parts of the tumor, or the segmentation may contain the rectangular or square tile at the same time. Alternatively, the rectangular or square tile and the segmentation of the tumor may overlap.

[0069] The inventors recognized that using a portion of a medical image corresponding to a tumor as input to a machine learning model gives more accurate results for similar patient searches because in many cases similar patients should be determined for a cancer disease and imaging features associated with the tumor are particularly indicative of the type of cancer disease.

[0070] According to another aspect of the present invention, the method comprises dividing the first medical image into a plurality of first tiles, wherein, in the step of determining the first feature vector, the first machine learning model is applied to one of the first tiles. In particular, the first machine learning model can be applied to the plurality of first tiles to determine a plurality of first feature vectors. In particular, the first machine learning model can be applied to all tiles in the first tiles to determine the first feature vectors of all tiles in the first tiles.

[0071] In particular, the tiles may be rectangular or square tiles. In particular, the tiles may be mutually exclusive (such that for each pixel of the first medical image, there is no more than one tile containing the pixel) and / or overlapping (such that for each pixel of the first medical image, there is at least one tile containing the pixel). In particular, all tiles may have the same size in terms of the number of pixels.

[0072] The inventors recognize that the first medical image typically includes a large number of pixels that cannot be processed with high accuracy by the machine learning model (particularly, for a full-slice image). By dividing the first medical image into tiles and applying the machine learning model to the tiles, the input data of the machine learning model can be reduced and better computational efficiency can be achieved. In addition, by calculating the feature vectors of the tiles, related similar tiles in the comparison image can be associated.

[0073] According to another possible aspect of the present invention, for each of the first tiles, a corresponding first feature vector is determined by applying the first machine learning model to the corresponding first tile. The inventors recognize that by calculating several first feature vectors for different tiles of the first medical image, similar medical images can be determined not based on the entire image, but for potentially more relevant sub-portions. In addition, several similar medical images for different portions of the first medical image can be found.

[0074] According to another possible aspect of the present invention, the first feature vector is based on the output of the first machine learning model when applied to each first tile in the subset of first tiles. In particular, the first feature vector may be an average or weighted average of the output of the first machine learning model when applied to each first tile in the subset of first tiles. The inventors recognized that by calculating the first feature vector based on multiple tiles, the overall impression of the first medical image and information from the entire first medical image can be included in the calculation of the first feature vector, wherein the entire first medical image does not need to be processed simultaneously, thereby improving computational efficiency.

[0075] According to another possible aspect of the invention, a subset of the first tiles is selected based on an attention mechanism of the first machine learning model. The inventors have recognized that using the attention mechanism results in the most relevant first tiles being included as a basis for calculating the first feature vector.

[0076] According to another aspect of the present invention, the similar medical image comprises a plurality of second image blocks, wherein one of the second image blocks is associated with a similar feature vector, and providing the similar medical image comprises displaying an association between one of the first image blocks and one of the second image blocks.

[0077] According to another possible aspect of the present invention, providing similar medical images includes displaying an association between the first image block and the second image block based on a similarity of feature vectors associated with the first image block and the second image block. In particular, the association may be displayed when the similarity satisfies a specific threshold.

[0078] Typically, an association between one of the first tiles and one of the second tiles is a (visual) indication that one of the first tiles is related to one of the second tiles. In particular, the association may be a visual indication that both one of the first tiles and one of the second tiles are marked in the same way (e.g. with the same border type or with the same symbol). Alternatively or additionally, the association may be a visual indication that one of the first tiles and one of the second tiles are connected (e.g. by a line). Alternatively or additionally, the association may be a visual indication based on a spatial relationship of one of the first tiles and one of the second tiles on a display (e.g. displaying one of the first tiles and one of the second tiles side by side).

[0079] The inventors realised that by displaying the associations, a user may easily identify related patches in the first medical image and the similar medical image that led to the selection of the similar medical image.

[0080] In a second aspect, the present invention relates to a computer-implemented method for providing a first machine learning model. The method includes receiving a training data set including training medical images and related non-imaging data. In addition, the method includes determining output data by applying a second machine learning model to the training medical images, wherein the second machine learning model includes the first machine learning model. The method also includes modifying parameters of the second machine learning model based on differences between the non-imaging data and the output data. The method also includes: extracting the first machine learning model from the second machine learning model; and providing the first machine learning model.

[0081] In particular, the difference between the non-imaging data and the output data may be a d-norm or a mean square error (corresponding to the sum of squared differences of elements in a vector). In particular, modifying the parameters of the second machine learning model may be based on minimizing the difference between the non-imaging data and the output data, for example by using a gradient descent method. In particular, in the case where the second machine learning model is a neural network, a back propagation algorithm may be used within the gradient descent method. In particular, the first machine learning model comprises the parameters of the second machine learning model.

[0082] In particular, extracting the first machine learning model from the second machine learning model (in the current case where the second machine learning model includes the first machine learning model) may correspond to trimming the second machine learning model to include only a portion also included in the first machine learning model. In the case where the first machine learning model and the second machine learning model are artificial neural networks, extracting the first machine learning model from the second machine learning model may correspond to using only the nodes and weights of the second machine learning model that are also included in the first machine learning model.

[0083] The inventors realized that, based on the proposed method, a first machine learning model can be trained for use in one of the methods according to the first aspect of the invention.

[0084] According to a third aspect, the present invention relates to an image providing system for providing similar medical images, comprising:

[0085] - means for receiving a first medical image associated with a first patient;

[0086] - means for determining a first feature vector by applying a first machine learning model to the first medical image,

[0087] wherein the first machine learning model is trained based on a training data set, each training data set comprising training medical images and related non-imaging data;

[0088] - means for receiving a plurality of second feature vectors,

[0089] wherein the plurality of second feature vectors are results obtained by applying the first machine learning model to the plurality of second medical images;

[0090] - means for determining similar feature vectors from a plurality of second feature vectors based on the first feature vector;

[0091] - means for selecting similar medical images from a plurality of second medical images, wherein similar medical images are associated with similar feature vectors; and

[0092] - Means for providing similar medical images.

[0093] According to a fourth aspect, the present invention relates to an image providing system for providing similar medical images, comprising a processor and an interface configured to:

[0094] - receiving a first medical image associated with a first patient;

[0095] - determining a first feature vector by applying a first machine learning model to the first medical image,

[0096] wherein the first machine learning model is trained based on a training data set, each training data set comprising training medical images and related non-imaging data;

[0097] - receiving a plurality of second feature vectors,

[0098] wherein the plurality of second feature vectors are results obtained by applying the first machine learning model to the plurality of second medical images;

[0099] - determining similar feature vectors from a plurality of second feature vectors based on the first feature vector;

[0100] - selecting similar medical images from the plurality of second medical images, wherein the similar medical images are associated with similar feature vectors; and

[0101] -Provide PROV-IMG-S similar medical images.

[0102] In particular, the image providing system may be configured to perform the method for providing similar medical images according to the present invention and aspects thereof. The image providing system is configured to perform the method and aspects thereof by means of its apparatus, interface and / or its processor configured to perform the corresponding method steps.

[0103] According to a fifth aspect, the invention relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method for providing similar medical images according to the invention and aspects thereof.

[0104] According to a sixth aspect, the present invention relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform a method for providing similar medical images according to the present invention and aspects thereof.

[0105] Implementing the invention or one of its aspects by means of a computer program product and / or a computer readable medium has the following advantage: existing servers, devices and clients can be easily adapted by software updates to work as proposed by the invention.

[0106] The computer program product may be, for example, a computer program, or may include another element in addition to the computer program. The other element may be hardware, such as a memory device on which the computer program is stored, a hardware key for using the computer program, etc., and / or software, such as a document or software key for using the computer program.

[0107] According to a seventh aspect, the present invention relates to a model providing system for providing a first machine learning model, comprising:

[0108] - means for receiving a training data set comprising training medical images and associated non-imaging data,

[0109] - means for determining output data by applying a second machine learning model to the training medical images, wherein the second machine learning model comprises the first machine learning model,

[0110] - means for modifying parameters of a second machine learning model based on differences between the non-imaging data and the output data,

[0111] - means for extracting the first machine learning model from the second machine learning model,

[0112] -Means for providing a first machine learning model.

[0113] According to an eighth aspect, the present invention relates to a model providing system for providing a first machine learning model, comprising a processor and an interface configured to:

[0114] - receiving a training dataset comprising training medical images and related non-imaging data,

[0115] - determining output data by applying a second machine learning model to the training medical images, wherein the second machine learning model comprises the first machine learning model,

[0116] - modifying parameters of a second machine learning model based on the difference between the non-imaging data and the output data,

[0117] - extracting the first machine learning model from the second machine learning model,

[0118] - Provide the first machine learning model.

[0119] In particular, the model providing system may be configured to perform the method for providing a first machine learning model according to the present invention and aspects thereof. The image providing system is configured to perform the method and aspects thereof by means of its apparatus, interface and / or its processor configured to perform the corresponding method steps.

[0120] According to a ninth aspect, the present invention relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method for providing a first machine learning model according to the present invention and aspects thereof.

[0121] According to a tenth aspect, the present invention relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform a method for providing a first machine learning model according to the present invention and aspects thereof.

[0122] Implementing the present invention or one of its aspects by means of a computer program product and / or a computer readable medium has the following advantage: existing servers, devices and clients can be easily adapted by software updates to work as proposed by the present invention.

[0123] The computer program product may be, for example, a computer program, or may include another element in addition to the computer program. The other element may be hardware, such as a memory device on which the computer program is stored, a hardware key for using the computer program, etc., and / or software, such as a document or software key for using the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] The characteristics, features and advantages of the invention described above and the manner in which they are achieved become clearer and more easily understood in view of the following description and embodiments that will be described in detail in the context of the accompanying drawings. The following description does not limit the invention to the embodiments included. In different drawings, the same components or parts may be marked with the same reference numerals. Generally, the drawings are not drawn to scale.

[0125] The numbering and / or order of the method steps is intended to facilitate understanding, and unless otherwise explicitly stated or implicitly made clear, the numbering and / or order of the method steps should not be interpreted as meaning that a specified step must be performed according to the number of its reference numerals and / or its order in the drawings. In particular, several or even all of the method steps may be performed simultaneously, in an overlapping manner or sequentially.

[0126] In the following:

[0127] Figure 1 shows a data flow diagram according to an embodiment of the present invention,

[0128] Figure 2 shows a data flow diagram according to another embodiment of the present invention,

[0129] Figure 3 shows a data flow diagram related to the training of a first machine learning model according to another embodiment of the present invention,

[0130] Figure 4 The structure of the first machine learning model and the second machine learning model according to an embodiment of the present invention is shown.

[0131] Figure 5 A flowchart of a method for providing similar medical images according to an embodiment of the present invention is shown.

[0132] Figure 6 A flowchart of a method for providing similar medical images according to another embodiment of the present invention is shown.

[0133] Figure 7 A flowchart of a method for providing a first machine learning model according to an embodiment of the present invention is shown.

[0134] Figure 8 An example of a first medical image being a whole-slice histopathology image is shown,

[0135] Fig. 9 showing an association between a first image block of a first medical image and a second image block of a similar medical image,

[0136] Fig.10 An image providing system according to an embodiment of the present invention is shown.

[0137] Fig.11 A model providing system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0138] Figure 1 A data flow diagram according to an embodiment of the present invention is shown.

[0139] In this embodiment, the first medical image IMG-1 is used directly as input to the first machine learning model ML-1, so that a first feature vector FV-1 is generated. Alternatively, the first medical image IMG-1 and additional input data, such as imaging metadata or patient demographic data, may be used. In this embodiment, the first medical image IMG-1 is a user-selected tile of 150×150 pixels extracted from a whole-slice pathology image stained with H&E. The first feature vector FV-1 is a 512-dimensional vector with real-valued elements.

[0140] In the same manner, each second medical image IMG-2 of the plurality of second medical images IMG-2 is used as an input to the first machine learning model ML-1, so that a plurality of second feature vectors FV-2 are generated. Alternatively, the second medical images IMG-2 and additional input data, such as imaging metadata or patient demographic data, may be used. In this embodiment, all second medical images IMG-2 are 150×150 pixel tiles extracted from the full slice image, and all second feature vectors FV-2 are 512-dimensional vectors with real-valued elements.

[0141] Furthermore, the first feature vector FV-1 is compared with each of the plurality of second feature vectors FV-2. In this embodiment, for each comparison, the sum of squared differences s(v, w i )=(v–w i )·(v–w i ) / 512 calculates the similarity metric, where “·” represents the scalar product / dot product between two vectors, and where v is the first eigenvector, and w i is one of the second eigenvectors FV-2. From all comparisons, select the one with the highest similarity (here, where the smallest s(v,w i )≥0) as the similar feature vector FV-S. Alternatively, instead of the sum of squared differences method described in Section 4 of the document "Detect-to-Retrieve: Efficient Regional Aggregation for Image Search" by Teichmann et al., arXiv:1812.01584v2, it can be used to compare the feature vectors FV-1, FV-2 and the feature vector set.

[0142] Based on the similar feature vector FV-S, a similar medical image IMG-S can be determined because the similar medical image IMG-S is an image used as an input to the first machine learning model ML-1 to calculate the similar feature vector FV-S. For example, there may be a database table that associates the second feature vector FV-2 with the second medical image IMG-2, which can be queried to determine the similar medical image IMG-S based on the similar feature vector FV-S.

[0143] Of course, several similar feature vectors FV-S and several corresponding similar medical images IMG-S may also be selected. For example, a predetermined number of second feature vectors FV-2 may be selected as similar feature vectors FV-S (e.g., second feature vectors FV-2 with the highest similarity), or all feature vectors FV-2- that meet a threshold similarity may be selected as similar feature vectors FV-S.

[0144] Figure 2 A data flow diagram according to another embodiment of the present invention is shown.

[0145] and Figure 1 In this embodiment, the first medical image IMG-1 is not used directly as input to the first machine learning model ML-1, but the first tile PTCH-1 from the first medical image IMG-1 is used directly as input to the first machine learning model ML-1. In this embodiment, the first medical image IMG-1 as a full slice image is divided into (possibly overlapping) first tiles PTCH-1 of 150×150 pixels. For each of the first tiles PTCH-1, a first feature vector FV-1 is determined by applying the first machine learning model ML-1, wherein the first feature vector FV-1 is a 512-dimensional vector with real-valued elements.

[0146] Furthermore, the plurality of second medical images IMG-2 are divided into tiles, and for each tile of each second medical image IMG-2, a second feature vector FV-2 is calculated based on applying the first machine learning model ML-1 to the corresponding tile.

[0147] In this embodiment, for each first feature vector FV-1, a similar feature vector FV-S is determined from a plurality of second feature vectors, such as for Figure 1As described. Alternatively, a similar feature vector FV-S may also be determined for a subset of the first feature vector FV-1 (for example, the first feature vector FV-1 whose corresponding block corresponds to the segmented portion of the tumor in the first medical image IMG-1). Alternatively, a pair including one of the first feature vectors FV-1 and one of the second feature vectors FV-2 having the highest mutual similarity measure may also be determined, and the determined second feature vector FV-2 may be selected as the similar feature vector FV-S.

[0148] Based on at least one similar feature vector FV-S, at least one corresponding similar medical image IMG-S can be selected. In the case where several of these similar feature vectors FV-S correspond to the same similar medical image IMG-S (different second tiles PTCH-S), the number of similar medical images IMG-S may be less than the number of at least one similar feature vector FV-S. Alternatively, at least one second tile PTCH-S corresponding to the similar feature vector FV-S can also be selected as the similar medical image IMG-S (i.e., the complete similar medical image IMG-S is not retrieved, but only the corresponding second tile PTCH-S is retrieved).

[0149] Figure 3 A data flow diagram related to the training of the first machine learning model ML-1 according to another embodiment of the present invention is shown.

[0150] In this embodiment, the first machine learning model ML-1 is part of the second machine learning model ML-2. In addition to the first machine learning model ML-1, the second machine learning model also includes a classifier unit CU. The second machine learning model ML-2 may also include additional components. In this embodiment, the second machine learning model ML-2 is a neural network including multiple layers, and the first machine learning model ML-1 includes a first subset of consecutive layers, and the classifier unit CU includes a second subset of consecutive layers.

[0151] During the training of the machine learning models ML-1, ML-2, a training data set is used, wherein each training data set comprises a training medical image IMG-T and associated non-imaging data PDAT-T. In particular, the training medical image IMG-T and the non-imaging data PDAT-T are associated with the same patient. In this embodiment, the training medical image IMG-T is a user-selected tile of 150×150 pixels extracted from a whole-slice pathology image stained with H&E. However, other types of training medical images, such as those recorded by a computed tomography device or a magnetic resonance imaging device, may also be used.

[0152] The training medical image IMT-T is used as input to the second machine learning model ML-2 and as input to the first machine learning model ML-1, because in this embodiment, the first machine learning model ML-1 is the first part of the processing direction of the second machine learning model ML-2 relative to the feedforward network. When applied to the training medical image IMT-T, the second machine learning model ML-2 creates output data OUT-T. Internally, the first machine learning model ML-1 calculates a feature vector, and the classifier unit CU takes the feature vector as input and calculates output data OUT-T as output.

[0153] Within the training, the output data OUT-T and the non-imaging data PDAT-T are compared, and based on the comparison, at least one parameter of the second machine learning model ML-2 is adjusted. In this embodiment, at least one parameter of the second machine learning model ML-2 is a parameter of the first machine learning model ML-1 and / or a parameter of the classifier unit CU.

[0154] In this embodiment, the non-imaging data PDAT-T includes genomic data and data related to a molecular subtype of the patient's colorectal cancer (typically, one of the categories "CMS1", "CMS2", "CMS3" and "CMS4" of "hypermutation subtype", "standard subtype", "metabolic subtype" and "mesenchymal subtype"). The genomic data refers to whether the patient's genome includes a mutation in the BRAF gene ("proto-oncogene B-Raf"), and specifically, whether the BRAF gene includes a BRAF V600E mutation. In particular, the non-imaging data PDAT-T can be a three-dimensional vector p, where p1∈{0,1} depends on whether there is a mutation in the patient's BRAF gene, p2∈{0,1} depends on whether the patient's genome includes a BRAF V600E mutation, and p3∈{1,2,3,4} depends on the molecular subtype of the patient's colorectal cancer.

[0155] In this embodiment, the output data OUT-T is also a three-dimensional vector with real numbers v1, v2∈[0,1] and v3∈[0,4]. In training, based on using the loss function L=(p1-v1) 2 +(p2-v2) 2 +(p3-v3) 2 The back-propagation algorithm is used to adjust the parameters of the second machine learning network ML-2, and the loss function is based on the difference between the non-imaging data PDAT-T and the output data OUT-T.

[0156] Figure 4 A first machine learning model ML-1 and a second machine learning model ML-2 used in one of the embodiments of the present invention are shown.

[0157] In this embodiment, the second machine learning model ML-2 includes a first machine learning model ML-1 and a classifier unit CU, wherein the first machine learning model ML-1 takes the first medical image IMG-1 as input and creates a first feature vector FV-1 as output, and the classifier unit CU takes the first feature vector FV-1 as input and creates output data OUT-T corresponding to the non-imaging data PDAT-T.

[0158] In this embodiment, the input image IMG-1 is a square tile of size 250×250 pixels selected from a stained whole-slice image. The first machine learning model ML-1 includes a feature unit ML-F and an attention unit ML-A, wherein the attention unit ML-A is based on an attention mechanism. The feature unit ML-F takes the first medical image IMG-1 as input and creates multiple sub-features as output. The attention unit ML-A takes multiple sub-features as input and calculates an attention score for each of the sub-features.

[0159] In this embodiment, the feature unit ML-F is derived from the well-known ResNet50 model (e.g., as described in "Deep Residual Learning for Image Recognition" by He et al., arXiv:1512.03385). However, since the feature unit ML-F should not generate a classification value, but a certain number of feature vectors, a certain number of the last layers of the ResNet50 model are skipped, so that multiple sub-features are generated. For example, there may be 49 sub-features (given by f n ), each sub-feature is a 512-dimensional vector.

[0160] In this embodiment, the attention unit ML-A includes an attention score function α(f n ,θ). Then, the output of the attention unit ML-A is the weighted sum Σ n α(f n ,θ)f n , and the parameter θ can be learned based on back propagation when the output of the classifier unit CU is compared with the ground truth which is the non-imaging data PDAT-T.

[0161] In this embodiment, the classifier unit CU is a fully connected network that takes 512 numerical values ​​(corresponding to the elements of the first feature vector FV-1) as input. In other words, the input layer of the classifier unit CU has 512 nodes. The number of output values ​​corresponds to the classification items that should be calculated by the second machine learning model ML-2. For example, in the case where the non-imaging data PDAT-T is a three-dimensional vector, the output includes at least three (preferably exactly three) numerical values, and the output layer of the classifier unit CU includes 3 nodes. In this case, there may be three hidden layers with 120, 40 and 15 nodes. Alternatively, the classifier unit may also be a convolutional neural network.

[0162] Figure 5 A flow chart of a method for providing a similar medical image IMG-S according to an embodiment is shown.

[0163] The first step of the method shown is to receive REC-IMG-1 a first medical image IMG-1 associated with a first patient PAT-1. In this embodiment, the first medical image IMG-1 is a rectangular tile of 250×250 pixels selected from a full-slice digital pathology image. Alternatively, other types of first medical images IMG-1 can also be used. For example, a DICOM image created in a radiological examination using magnetic resonance imaging or computer tomography can be used. The step of receiving REC-IMG-1 is performed by the interface PSYS.IF of the image providing system PSYS.

[0164] The next step of the method shown is to determine a first feature vector FV-1 DET-FV-1 by applying a first machine learning model ML-1 to the first medical image IMG-1 by the computing unit PSYS.CU of the image providing system PSYS. The machine learning model ML-1 has the following characteristics: Figure 4 Displayed and targeted Figure 4 The structure described. The first machine learning model ML-1 is trained based on a training data set, each training data set comprising training medical images IMG-T and related non-imaging data PDAT-T. The training medical images IMG-T are tiles of size 250×250 pixels of a full slice image, in particular, do not contain medical images related to the first patient PAT-1. Alternatively, other types of training medical images IMG.T may be used, however, advantageously, the size and type of the training medical images IMG-T should match the size and type of the first medical images IMG-1. In this embodiment, the non-imaging data PDAT-T has the following structure: Figure 3The described structure may alternatively also use another non-imaging data PDAT-T. In this embodiment, the training medical images IMG-T and the non-imaging data PDAT-T in each training data set correspond to the same patient.

[0165] Figure 5 The next step of the embodiment shown in is to receive REC-FV-2 multiple second feature vectors FV-2 through the interface PSYS.IF of the image providing system PSYS, wherein the multiple second feature vectors FV-2 are the result of applying the first machine learning model ML-1 to multiple second medical images IMG-2. In this embodiment, the second medical image IMG-2 is a tile of size 250×250 pixels of the full slice image. Alternatively, other types of second medical images IMG-2 can be used, however, advantageously, the size and type of the second medical image IMG-2 should match the size and type of the first medical image IMG-1.

[0166] In this embodiment, each of the second feature vectors FV-2 has the same structure and the same dimension as the first feature vector FV-1. In particular, each of the second feature vectors FV-2 is a 512-dimensional vector with real-valued elements. In this embodiment, each of the second feature vectors FV-2 is the result of applying the first machine learning model ML-1 to exactly one of the second medical images IMG-2, in other words, there is a bijective mapping between the second medical image IMG-2 and the second feature vector FV-2.

[0167] The next step of the embodiment is to determine a DET-FV-S similar feature vector FV-S from a plurality of second feature vectors FV-2 based on the first feature vector FV-1 and using the computing unit PSYS.CU of the image providing system. The similar feature vector FV-S is one of the second feature vectors FV-2. In particular, according to a predefined metric, the similar feature vector FV-2 is a second feature vector FV-2 among the second feature vectors FV-2 that is closest to the first feature vector FV-1. In this embodiment, the Euclidean distance is used as a metric, so that the similar feature vector FV-S is a second feature vector FV-2 among the second feature vectors FV-2 that has the smallest Euclidean distance from the first feature vector FV-1. Alternatively, other metrics may be used, such as the Manhattan metric or the cosine similarity.

[0168] Figure 5The next step of the method shown in is to select SEL similar medical images IMG-S from multiple second medical images IMG-2, wherein the similar medical images IMG-S are associated with similar feature vectors FV-S. In this embodiment, the similar medical images IMG-S are associated with the similar feature vectors FV-S by the similar feature vectors FV-S that have been calculated by applying the first machine learning model ML-1 to the similar medical images IMG-S. In particular, the relationship between the second medical image IMG-2 and the second feature vectors FV-2 can be stored in a database, so that selecting SEL similar medical images IMG-S can include accessing or querying the database to find similar medical images IMG-S associated with the similar feature vectors FV-S.

[0169] The last step of the illustrated embodiment is to provide the PROV-IMF-S similar medical image IMG-S using the interface PSYS.IF of the image providing system PSYS. In particular, the step of providing the PROV-IMG-S similar medical image IMG-S may include: transmitting the similar medical image IMG-S to another entity; storing the similar medical image IMG-S in a storage unit and / or displaying the similar medical image IMG-S on a display unit.

[0170] Figure 5 A flow chart of a method for providing a similar medical image IMG-S according to another embodiment is shown. In this embodiment, the steps of receiving a first medical image IMG-1 REC-IMG-1, receiving a plurality of second feature vectors REC-FV-2, determining a similar feature vector FV-S DET-FV-S, selecting a similar medical image IMG-S SEL-IMG-S, and providing a similar medical image IMG-S PROV-IMG-S may include the same steps as for example Figure 5 The same advantageous features and options as described for the embodiments of the present invention.

[0171] In this embodiment, the method further comprises an optional step of determining, by a computing unit PSYS.CU of the image providing system PSYS, a segmentation of the DET-SEM tumor within the first medical image IMG-1. The segmentation can be determined in various ways, for example, a neural network with a U-Net structure can be used which takes as input the first medical image IMG-1 and calculates as output a binary mask of the same size, wherein pixels with a value of 1 correspond to tumor pixels and pixels with a value of 0 correspond to non-tumor pixels. Alternatively, other segmentation methods which are not based on a learning algorithm can also be used.

[0172] In this embodiment, the method further comprises an optional step of dividing the first medical image IMG-1 DIV-IMG-1 into a plurality of first image tiles PTCH-1 by a computing unit PSYS.CU of the image providing system PSYS. In this embodiment, a maximum number of non-overlapping first image tiles PTCH-1 of a predefined pixel size (e.g., 250×250 pixels) is determined. Alternatively, a predefined number M·N (where M, N are predefined integers) of first image tiles PTCH-1 may be created by equally dividing the first image IMG-1 M times along the first axis and equally dividing the first image IMG-1 N times along the second axis.

[0173] The step of determining the segmentation of the DET-SEG tumour may be performed before the step of segmenting the DIV-IMG-1 first medical image IMG-1 and the step of determining the segmentation of the DET-SEG tumour may be performed before the step of segmenting the DIV-IMG-1 first medical image IMG-1. Alternatively, both steps may be performed in parallel.

[0174] In this embodiment, within the step of determining the DET-FV-1 first feature vector FV-1, the first machine learning model ML-1 is applied to at least one of the first tiles PTCH-1 corresponding to the segmented tumor (i.e., by having an overlap with the segmented tumor mask). In particular, the first machine learning model ML-1 is applied to all tiles in the first tiles PTCH-1 corresponding to the segmented tumor, and separate first feature vectors FV-1 are calculated for all tiles in these first tiles PTCH-1. In particular, this means that the step of determining the DET-FV-1 first feature vector is performed for each of the relevant first tiles PTCH-1, and that a similar feature vector FV-S and a similar medical image IMG-S are determined for each of the relevant first tiles PTCH-S. Alternatively, a single similar feature vector FV-S and a single similar medical image IMG-S can be determined based only on the feature vector from the plurality of second feature vectors FV-2 having the greatest similarity to any one of the first feature vectors FV-1.

[0175] Furthermore, in this embodiment, the step of determining the first feature vector FV-1 of DET-FV-1 comprises: determining an optional subset of a plurality of sub-features of DET-SF by applying a feature unit ML-F to the first medical image IMG-1; and determining an optional subset of a first feature vector FV-1 of DET-AM by applying an attention unit ML-A to the plurality of sub-features, such as for Figure 4 As described.

[0176] Furthermore, the step of DET-FV-1 the first feature vector FV-1 comprises reducing the dimensionality of an optional subset of the RED-DIM sub-features by principal component analysis and / or applying a learnable bottleneck layer.

[0177] Figure 7 A flowchart of a method for providing a first machine learning model ML-1 according to an embodiment of the present invention is shown.

[0178] According to an embodiment, the method comprises as a first step receiving a T-REC training data set comprising training medical images IMG-T and associated non-imaging data PDAT-T. In particular, the training medical images IMG-T and the non-imaging data PDAT-T are related by being acquired from the same patient. The step of receiving the T-REC training data set is performed by an interface TSYS.IF of the model providing system TSYS.

[0179] The next step of the illustrated embodiment is to determine T-DET output data by applying a second machine learning model ML-2 to the training medical image IMG-T, wherein the second machine learning model ML-2 includes the first machine learning model ML-1. In particular, the first machine learning model ML-1 and the second machine learning model ML-2 have the same characteristics as for Figure 4 The described structure enables the second machine learning model ML-2 to include the first machine learning model ML-1 and the classifier unit CU. The step of determining the T-DET output data is performed by the computing unit TSYS.CU of the model providing system TSYS.

[0180] The next step of the illustrated embodiment is to modify the parameters of the T-MDF second machine learning model ML-2 based on the difference between the non-imaging data PDAT-T and the output data. In particular, the parameters of the second machine learning model ML-2 correspond to the weights of the edges within the neural network corresponding to the second machine learning model ML-2. In particular, the parameters of the second machine learning model ML-2 are also the parameters of the first machine learning model ML-1 at the same time. In particular, the parameters are modified based on a gradient descent method, in particular based on a back-propagation algorithm. The step of modifying the parameters of the T-MDF second machine learning model ML-2 is performed by a computing unit TSYS.CU of the model providing system TSYS.

[0181] The next step of the illustrated embodiment is to extract the T-EXT first machine learning model ML-1 from the second machine learning model ML-2. This step is performed by the computing unit TSYS.CU of the model providing system TSYS. In this embodiment, the first machine learning model ML-1 is a sub-part of the second machine learning model ML-2, and extracting the T-EXT first machine learning model ML-1 can correspond to deleting all parts that are related to the second machine learning model ML-2 but not to the first machine learning model ML-1.

[0182] The last step of the illustrated embodiment is to provide T-PROV the first machine learning model ML-1, which is performed by the interface TSYS.IF of the model providing system TSYS.

[0183] The steps as described above correspond to a single step of training the first machine learning model ML-1 and the second machine learning model ML-2. In order to improve the performance of the first machine learning model ML-1, these steps should be performed several times for different training data sets including training medical images IMG-T and related non-imaging data PDAT-T.

[0184] In this embodiment, a set of 1000 training data sets associated with 1000 different patients is used. For each of the training data sets, the non-imaging data PDAT-T is a three-dimensional vector p, where p1∈{0,1} depends on whether there is a mutation in the patient's BRAF gene, p2∈{0,1} depends on whether the patient's genome includes a BRAF V600E mutation, and p3∈{1,2,3,4} depends on the molecular subtype of the patient's colorectal cancer. All of the values ​​can be determined by determining the gene panel of the corresponding patient. In addition, for each of the training data sets, the training medical image IMG-T is a user-selected tile of 150×150 pixels extracted from a whole-slice pathology image stained with H&E of the corresponding patient. Multiple tiles can also be selected from a single whole-slice pathology image to increase the number of training data sets.

[0185] Figure 8 An example of a first medical image IMG-1 as a two-dimensional stained whole-slice image with N×N pixels is shown. In this embodiment, the first medical image IMG-1 is divided into 36 square first tiles PTCH-1, each of which is a two-dimensional pixel image with N / 6×N / 6 pixels.

[0186] The first medical image IMG-1 is based on a tissue sample obtained from a patient by means of a biopsy. To prepare a tissue section from the tissue sample, a section is cut from the tissue sample with a punching tool (also referred to as a "punch biopsy" or "block"), wherein the section is cut into micrometer-thick sections (tissue sections). The preparation of the tissue section also includes staining the tissue section with H&E (abbreviation for "hematoxylin and eosin") staining.

[0187] To create the first medical image IMG-1 as a whole-slice image, the tissue slice stained with H&E stain is scanned with a dedicated histopathology scanner, which scans the entire tissue slice mounted on an object carrier and converts it into a color pixel image. The first medical image IMG-1 as a whole-slice image can then be digitally processed and, in particular, can be archived in a suitable database.

[0188] Fig. 9 An example of a first medical image IMG-1 and a corresponding similar medical image IMG-S as determined by one of the methods according to embodiments of the present invention is shown.

[0189] In this embodiment, the first medical image IMG-1 includes a plurality of first tiles PTCH-1, and the similar medical image IMG-S includes a plurality of second tiles PTCH-S, wherein each of the tiles is two-dimensional and includes the same number of pixels.

[0190] Selecting a similar medical image IMG-S from a plurality of second medical images IMG-2, wherein each of the second medical images IMG-2 is obtained by Figure 8 The described method creates a whole slice image of a tissue sample. The selection of similar medical images IMG-S is based on the similarity of a first feature vector FV-1 determined based on a tile of the first medical image IMG-1 and a second feature vector FV-2 determined based on a tile of the second medical image IMG-2.

[0191] Furthermore, in this embodiment, associations ASSOC.1, ASSOC.2, ASSOC.3 between the tiles PTCH-1.1, PTCH-1.2, PTCH-1.3 of the first tile PTCH-1 and the tiles PTCH-S.1, PTCH-S.2, PTCH-S.3 of the second tile PTCH-S are shown. In this embodiment, the associations are shown by displaying arrows connecting the corresponding tiles PTCH-1.1, PTCH-1.2, PTCH-1.3 of the first tile PTCH-1 and the corresponding tiles PTCH-S.1, PTCH-S.2, PTCH-S.3 of the second tile PTCH-S.

[0192] In particular, in this embodiment, a similar medical image IMG-S is selected from the plurality of second medical images IMG-2 based on the highest similarity between a first feature vector FV-1 associated with a first tile PTCH-1.1 among the plurality of first tiles PTCH-1 and a similar feature vector FV.S associated with a first tile PTCH-S.1 among the plurality of second tiles PTCH-2. Therefore, an association ASSOC.1 is displayed between the first tile PTCH-1.1 among the plurality of first tiles PTCH-1 and the first tile PTCH-S.1 among the plurality of second tiles PTCH-2.

[0193] Furthermore, in this embodiment, additional associations ASSOC.2, ASSOC.3 are shown. These additional associations ASSOC.2, ASSOC.3 correspond to pairs consisting of tiles PTCH-1.2, PTCH-1.3 of the plurality of first tiles PTCH-1 and tiles PTCH-S.2, PTCH-S.3 of the plurality of second tiles PTCH-2, wherein the similarity between the associated feature vectors is above a predefined threshold.

[0194] Fig.10 shows an embodiment of an image providing system PSYS, Fig.11 An embodiment of a model providing system TSYS is shown. The image providing system PSYS is configured to perform a method for providing a similar medical image IMG-S according to an embodiment of the present invention. The model providing system TSYS is configured to perform a method for providing a first machine learning model ML-1 according to an embodiment of the present invention.

[0195] The image providing system PSYS and / or the model providing system TSYS may be a (personal) computer, a workstation, a virtual machine running on host hardware, a microcontroller or an integrated circuit. In particular, the image providing system PSYS and / or the model providing system TSYS may be a mobile device, such as a smartphone or a tablet computer. As an alternative, the image providing system PSYS and / or the model providing system TSYS may be a real or virtual group of computers (the technical term for a real group of computers is a "cluster", the technical term for a virtual group of computers is a "cloud").

[0196] Each of the image providing system PSYS and / or the model providing system TSYS may include an interface PSYS.IF, TSYS.IF, a computing unit PSYS.CU, TSYS.CU and a memory unit PSYS.CU, TSYS.CU. The interface PSYS.IF, TSYS.IF may be a hardware interface or as a software interface (e.g., PCIBus, USB or FireWire). The computing unit PSYS.CU, TSYS.CU may include hardware elements and software elements, such as a microprocessor, a CPU (abbreviation for “central processing unit”), a GPU (abbreviation for “graphics processing unit”), a field programmable gate array (abbreviated as “FPGA”) or an ASIC (abbreviation for “application-specific integrated circuit”). The computing unit PSYS.CU, TSYS.CU may be configured for multithreading, i.e., the computing unit may host different computing processes at the same time, execute in parallel or switch between active computing processes and passive computing processes. Each of the interfaces PSYS.IF, TSYS.IF, the calculation units PSYS.CU, TSYS.CU and the memory units PSYS.CU, TSYS.CU may include several sub-units configured to perform different tasks and / or be spatially separated.

[0197] The image providing system PSYS is connected to a database DB via a network NETW. In particular, the database DB stores a plurality of second feature vectors FV-2 and / or a plurality of second medical images IMG-2. The network NETW can be implemented as a LAN (abbreviation for "local area network"), in particular a WiFi network, or any other local connection. Alternatively, the network NETW can be the Internet. In particular, the network NETW can be implemented as a VPN (abbreviation for "virtual private network"). Alternatively, the database DB can also be integrated into the image providing system PSYS, for example, the database DB can be stored in a memory unit PSYS.MU of the image providing system PSYS. In this case, the database is connected via an internal connection.

[0198] The following terms are also part of this disclosure:

[0199] Clause 1.1: A computer-implemented method for providing a similar medical image IMG-S, comprising:

[0200] - receiving REC-IMG-1 and a first medical image IMG-1 associated with a first patient PAT-1;

[0201] - determining a first feature vector FV-1 DET-FV-1 by applying a first machine learning model ML-1 to the first medical image IMG-1,

[0202] Wherein, the first machine learning model ML-1 is trained based on a training data set, each training data set including a training medical image IMG-T and related non-imaging data PDAT-T;

[0203] - receiving REC-FV-2 a plurality of second feature vectors FV-2,

[0204] The plurality of second feature vectors FV-2 are the results obtained by applying the first machine learning model ML-1 to the plurality of second medical images IMG-2;

[0205] - determining a DET-FV-S similar feature vector FV-S from a plurality of second feature vectors FV-2 based on the first feature vector FV-1;

[0206] - selecting SEL-IMG-S similar medical images IMG-S from the plurality of second medical images IMG-2, wherein the similar medical images IMG-S are associated with similar feature vectors FV-S; and

[0207] - Provides PROV-IMG-S similar medical images IMG-S.

[0208] Clause 1.2: A method according to clause 1.1, wherein each second feature vector FV-2 of the plurality of second feature vectors FV-2 is the result of applying the first machine learning model ML-1 to exactly one second medical image IMG-2 of the plurality of second medical images IMG-2.

[0209] Clause 1.3: The method according to clause 1.2, wherein there is a one-to-one correspondence between the plurality of second feature vectors FV-2 and the plurality of second medical images IMG-2.

[0210] Clause 1.4: The method according to one of clauses 1.1 to 1.3, wherein the non-imaging data PDAT-T has a different structure than the first feature vector FV-1.

[0211] Clause 1.5: The method of clause 1.4, wherein the non-imaging data PDAT-T has a lower dimension than the first eigenvector FV-1.

[0212] Clause 1.6: The method according to one of clauses 1.1 to 1.5, wherein the first medical image IMG-1 is a histopathology slice image, in particular a whole slice image.

[0213] Clause 1.7: The method of any of clauses 1.1 to 1.6, wherein the non-imaging data PDAT-T comprises at least one of the following categories: omics data, diagnostic data, treatment history, and demographic data.

[0214] Clause 1.8: The method of clause 1.7, wherein the non-imaging data PDAT-T comprises omics data, wherein the omics data comprises at least one of genomic data, transcriptomic data, proteomic data, and metabolomic data.

[0215] Clause 1.9: The method of any of clauses 1.6 to 1.8, wherein the non-imaging data comprises omics data, wherein the training medical image IMG-T shows a tumor of the patient, and wherein the omics data relates to a type of cancer of the patient.

[0216] Clause 1.10: The method according to clause 1.9, wherein the training medical image is a histopathological slice image showing tumor cells of a patient, in particular a whole slice image showing tumor cells of a patient.

[0217] Clause 1.11: A method according to one of clauses 1.1 to 1.10, wherein the first machine learning model ML-1 is trained by training a second machine learning model ML-2 comprising the first machine learning model ML-1, wherein the second machine learning model ML-2 is trained to predict non-imaging data PDAT-T based on the medical image IMG-T.

[0218] Clause 1.12: The method of clause 1.11, wherein the second machine learning model ML-2 is a classifier.

[0219] Clause 1.13: A method according to clause 1.11 or 1.12, wherein the second machine learning model ML-2 comprises the first machine learning model ML-1 and a classification unit CU, wherein the classification unit CU processes the output of the first machine learning model ML-1 and determines a classification value.

[0220] Clause 1.14: A method according to clause 1.13, wherein the classification unit CU comprises or consists of an additional neural network layer.

[0221] Clause 1.15: A method according to one of clauses 1.11 to 1.14, wherein at least one parameter of the second machine learning network ML-2 is based on a difference between an output of the second machine learning network ML-2 when the second machine learning network is applied to a training medical image and non-imaging data corresponding to the training medical image.

[0222] Clause 1.16: A method according to one of clauses 1.1 to 1.10, wherein output data is generated by applying a first machine learning network ML-1 to medical images IMG-1, IMG-2, IMG-T, the output data comprising feature vectors FV-1, FV-2 and non-imaging data PDAT.

[0223] Clause 1.17: A method according to any one of clauses 1.1 to 1.17, wherein the first machine learning model ML-1 comprises a feature unit ML-F and an attention unit ML-A, wherein the attention unit ML-A is based on an attention mechanism, and wherein the step of determining the first feature vector FV-1 of DET-FV-1 comprises:

[0224] --determine a plurality of sub-features of DET-SF by applying the feature unit ML-F to the first medical image IMG-1, and

[0225] --Determine the DET-AM first feature vector FV-1 by applying the attention unit ML-A to multiple sub-features.

[0226] Clause 1.18: A method according to clause 1.17, wherein the step of determining the first eigenvector FV-1 of DET-FV-1 comprises:

[0227] -Reduce the dimensionality of RED-DIM sub-features by principal component analysis and / or applying a learnable bottleneck layer.

[0228] Clause 1.19: A method according to clause 1.17 or clause 1.18, wherein the first machine learning model ML-1 is a deep local feature network.

[0229] Clause 1.20: A method according to any of Clauses 1.1 to 1.19, further comprising:

[0230] - determining a DET-SEG tumor segmentation in the first medical image IMG-1,

[0231] Therein, in the step of determining the first feature vector FV-1 of DET-FV-1, the machine learning model is applied to a portion of the first medical image IMG-1 corresponding to the segmented tumor.

[0232] Clause 1.21: A method in accordance with any of Clauses 1.1 to 1.19, including:

[0233] - dividing the first medical image IMG-1 DIV-IMG into a plurality of first picture tiles PTCH-1,

[0234] Therein, in the step of determining the first feature vector FV-1 of DET-FV-1, the first machine learning model ML-1 is applied to one of the first tiles PTCH-1.

[0235] Clause 1.22: A method according to clause 1.21, wherein, for each of the first tiles PTCH-1, the corresponding first feature vector FV-1 is determined by applying the first machine learning model ML-1 to the respective first tile PTCH-1.

[0236] Clause 1.23: A method according to clause 1.21, wherein the first feature vector FV-1 is based on the output of the first machine learning model ML-1 when applied to each first tile from the subset of first tiles PTCH-1.

[0237] Clause 1.24: A method according to clause 1.23, wherein the subset of the first tile PTCH-1 is selected based on an attention mechanism of the first machine learning model ML-1.

[0238] Clause 1.25: A method according to clause 1.21 or 1.22, wherein the similar medical image IMG-S comprises a plurality of second tiles PTCH-S, wherein one of the second tiles PTCH-S is associated with a similar feature vector FV-S, and wherein providing PROV-IMG-S the similar medical image IMG-S comprises displaying an association between one of the first tiles PTCH-1 and one of the second tiles PTCH-S.

[0239] Clause 1.26: A method according to clause 1.25, wherein providing the PROV-IMG-S similar medical image IMG-S includes displaying an association between the first tile PTCH-1 and the second tile PTCH-S based on the similarity of feature vectors FV-1, FV-S associated with the first tile PTCH-1 and the second tile PTCH-S.

[0240] Clause 2.1: A computer-implemented method for providing a first machine learning model ML-1, comprising:

[0241] - receiving a T-REC training dataset comprising training medical images IMG-T and associated non-imaging data PDAT-T,

[0242] - determining T-DET output data by applying a second machine learning model ML-2 to the training medical image IMG-T, wherein the second machine learning model ML-2 comprises the first machine learning model ML-1,

[0243] - modifying the parameters of the T-MDF second machine learning model ML-2 based on the difference between the non-imaging data PDAT-T and the output data,

[0244] - extract T-EXT from the second machine learning model ML-2, the first machine learning model ML-1,

[0245] - Provide T-PROV's first machine learning model ML-1.

[0246] Clause 2.2: A method according to clause 2.1, wherein the first machine learning model ML-1 includes parameters of the second machine learning model ML-2.

[0247] Clause 2.3: A method according to clause 2.1 or clause 2.2, wherein modifying the parameters of the second machine learning model ML-2 is based on a gradient descent method, in particular based on back propagation.

[0248] Clause 3.1: An image providing system PSYS for providing similar medical images IMG-S, comprising,

[0249] - means for receiving a first medical image IMG-1 associated with a first patient PAT-1 by REC;

[0250] - means for determining a first feature vector FV-1 of DET-FV-1 by applying a first machine learning model ML-1 to a first medical image IMG-1,

[0251] Wherein, the first machine learning model ML-1 is trained based on a training data set, each training data set including a training medical image IMG-T and related non-imaging data PDAT-T;

[0252] - means for receiving a plurality of second feature vectors FV-2 from REC-FV-2,

[0253] The plurality of second feature vectors FV-2 are the results obtained by applying the first machine learning model ML-1 to the plurality of second medical images IMG-2;

[0254] - means for determining a DET-FV-S similar feature vector FV-S from a plurality of second feature vectors FV-2 based on the first feature vector FV-1;

[0255] - means for selecting SEL-IMG-S similar medical images IMG-S from a plurality of second medical images IMG-2, wherein the similar medical images IMG-S are associated with similar feature vectors FV-S; and

[0256] - means for providing PROV-IMG-S with a similar medical image IMG-S.

[0257] Clause 3.2: An image providing system PSYS for providing a similar medical image IMG-S, comprising a processor PSYS.CU and an interface PSYS.IF configured to:

[0258] - receiving REC a first medical image IMG-1 associated with a first patient PAT-1;

[0259] - determining a first feature vector FV-1 DET-FV-1 by applying a first machine learning model ML-1 to the first medical image IMG-1,

[0260] Wherein, the first machine learning model ML-1 is trained based on a training data set, each training data set including a training medical image IMG-T and related non-imaging data PDAT-T;

[0261] - receiving REC-FV-2 a plurality of second feature vectors FV-2,

[0262] The plurality of second feature vectors FV-2 are the results obtained by applying the first machine learning model ML-1 to the plurality of second medical images IMG-2;

[0263] - determining a DET-FV-S similar feature vector FV-S from a plurality of second feature vectors FV-2 based on the first feature vector FV-1;

[0264] - selecting SEL-IMG-S similar medical images IMG-S from the plurality of second medical images IMG-2, wherein the similar medical images IMG-S are associated with similar feature vectors FV-S; and

[0265] - Provides PROV-IMG-S similar medical images IMG-S.

[0266] Clause 3.3: The image providing system SYS according to clause 3.1 or clause 3.2, comprising an additional device for performing a method according to one of clauses 1.1 to 1.26 or configured to perform a method according to one of clauses 1.1 to 1.26.

[0267] Clause 3.4: A computer program product comprising instructions which, when executed by a computer, cause the computer to perform a method according to one of clauses 1.1 to 1.26.

[0268] Clause 3.5: A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform a method according to one of clauses 1.1 to 1.26.

[0269] Clause 4.1: A model providing system TSYS for providing a first machine learning model ML-1, comprising:

[0270] - means for receiving a T-REC training data set comprising training medical images IMG-T and associated non-imaging data PDAT-T,

[0271] - means for determining T-DET output data by applying a second machine learning model ML-1 to the training medical image IMG-T, wherein the second machine learning model ML-2 comprises the first machine learning model ML-1,

[0272] - means for modifying parameters of the T-MDF second machine learning model ML-2 based on the difference between the non-imaging data PDAT-T and the output data,

[0273] - means for extracting the T-EXT first machine learning model ML-1 from the second machine learning model ML-2,

[0274] - Means for providing T-PROV a first machine learning model ML-1.

[0275] Clause 4.2: A model providing system TSYS for providing a first machine learning model ML-1, comprising a processor TSYS.CU and an interface TSYS.IF configured to:

[0276] - receiving a T-REC training dataset comprising training medical images IMG-T and associated non-imaging data PDAT-T,

[0277] - determining T-DET output data by applying a second machine learning model ML-1 to the training medical image IMG-T, wherein the second machine learning model ML-2 comprises the first machine learning model ML-1,

[0278] - modifying the parameters of the T-MDF second machine learning model ML-2 based on the difference between the non-imaging data PDAT-T and the output data,

[0279] - Extract T-EXT from the second machine learning model ML-2, the first machine learning model ML-1,

[0280] - Provide T-PROV's first machine learning model ML-1.

[0281] Clause 4.3: A system SYS is provided according to the model of clause 4.1 or clause 4.2, comprising additional means for carrying out the method according to one of clauses 2.1 to 2.3 or configured to carry out the method according to one of clauses 2.1 to 2.3.

[0282] Clause 4.4: A computer program product comprising instructions which, when executed by a computer, cause the computer to perform a method according to one of clauses 2.1 to 2.3.

[0283] Clause 4.5. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform a method according to any one of clauses 2.1 to 2.3.

[0284] In the absence of explicit description, the various embodiments or their various aspects and features can be combined or exchanged with each other without limiting or expanding the scope of the described invention, as long as such combination or exchange is meaningful and in the sense of the present invention. Where applicable, the advantages described for one embodiment of the present invention are also beneficial for other embodiments of the present invention.

Claims

1. A computer-implemented method for providing a similar medical image (IMG-S), comprising: - receiving (REC-IMG-1) a first medical image (IMG-1) associated with a first patient (PAT-1); - determining (DET-FV-1) a first feature vector (FV-1) by applying a first machine learning model (ML-1) to said first medical image (IMG-1), wherein the first machine learning model (ML-1) is trained based on a training data set, each training data set comprising a training medical image (IMG-T) and related non-imaging data (PDAT-T); - receiving (REC-FV-2) a plurality of second feature vectors (FV-2), wherein the plurality of second feature vectors (FV-2) are the result of applying the first machine learning model (ML-1) to a plurality of second medical images (IMG-2), - determining (DET-FV-S) a similar feature vector (FV-S) from said plurality of second feature vectors (FV-2) based on said first feature vector (FV-1); - selecting (SEL-IMG-S) the similar medical image (IMG-S) from the plurality of second medical images (IMG-2), wherein the similar medical image (IMG-S) is associated with the similar feature vector (FV-S); and - providing (PROV-IMG-S) said similar medical image (IMG-S).

2. The method according to claim 1, wherein: The first medical image (IMG-1) is a histopathology slice image, in particular a whole slice image.

3. The method according to claim 1 or 2, wherein: The non-imaging data (PDAT-T) includes at least one of the following categories: -omics data, - diagnostic data, - Treatment history, and - Demographic data.

4. The method according to claim 3, wherein: The non-imaging data (PDAT-T) includes omics data, wherein the omics data includes at least one of the following data: - genomic data, - transcriptomics data, - Proteomic data, and - Metabolomics data.

5. The method according to one of the preceding claims, wherein: The first machine learning model (ML-1) is trained by training a second machine learning model (ML-2) comprising the first machine learning model (ML-1), wherein the second machine learning model (ML-2) is trained to predict the non-imaging data based on medical images, wherein, advantageously, the second machine learning model (ML-2) is a classifier.

6. The method according to one of the preceding claims, wherein: The first machine learning model (ML-1) comprises a feature unit (ML-F) and an attention unit (ML-A), wherein the attention unit (ML-A) is based on an attention mechanism. The step of determining (DET-FV-1) the first feature vector (FV-1) comprises: - determining (DET-SF) a plurality of sub-features by applying said feature unit (ML-F) to said first medical image (IMG-1), and --Determine (DET-AM) the first feature vector (FV-1) by applying the attention unit (ML-A) to the plurality of sub-features.

7. The method according to claim 6, wherein: The step of determining (DET-FV-1) said first feature vector (FV-1) comprises reducing (RED-DIM) the dimensionality of the sub-features by principal component analysis and / or applying a learnable bottleneck layer.

8. The method according to claim 6 or 7, wherein: The first machine learning model (ML-1) is a deep local feature network.

9. The method according to one of the preceding claims, further comprising: - determining (DET-SEG) a segmentation of the tumor in said first medical image (IMG-1), Therein, in the step of determining (DET-FV-1) the first feature vector (FV-1), a machine learning model is applied to the part of the first medical image (IMG-1) corresponding to the segmented tumor.

10. The method according to one of claims 1 to 8, comprising: - dividing (DIV-IMG-1) the first medical image (IMG-1) into a plurality of first image tiles (PTCH-1), Therein, in the step of determining (DET-FV-1) the first feature vector (FV-1), the first machine learning model (ML-1) is applied to one of the first tiles (PTCH-1).

11. The method according to claim 10, wherein: The similar medical image (IMG-S) comprises a plurality of second tiles (PTCH-S), wherein one of the second tiles (PTCH-S) is associated with the similar feature vector (FV-S), and wherein providing (PROV-IMG-S) the similar medical image (IMG-S) comprises displaying an association between one of the first tiles (PTCH-1) and one of the second tiles (PTCH-S).

12. A computer-implemented method for providing a first machine learning model (ML-1), comprising: - receiving (T-REC) a training data set, said training data set comprising training medical images (IMG-T) and associated non-imaging data (PDAT-T), - determining (T-DET) output data (OUT-T) by applying a second machine learning model (ML-2) to the training medical image (IMG-T), wherein the second machine learning model (ML-2) includes the first machine learning model (ML-1), - modifying (T-MDF) parameters of said second machine learning model (ML-2) based on the difference between said non-imaging data (PDAT-T) and said output data (OUT-T), - extracting (T-EXT) said first machine learning model (ML-1) from said second machine learning model (ML-2), -Provide (T-PROV) the first machine learning model (ML-1).

13. An image providing system (PSYS) for providing a similar medical image (IMG-S), comprising: - means for receiving (REC) a first medical image (IMG-1) associated with a first patient (PAT-1); - means for determining (DET-FV-1) a first feature vector (FV-1) by applying a first machine learning model (ML-1) to said first medical image (IMG-1), wherein the first machine learning model (ML-1) is trained based on a training data set, each training data set comprising a training medical image (IMG-T) and related non-imaging data (PDAT-T); - means for receiving (REC-FV-2) a plurality of second feature vectors (FV-2), wherein the plurality of second feature vectors (FV-2) are results obtained by applying the first machine learning model (ML-1) to a plurality of second medical images (IMG-2); - means for determining (DET-FV-S) similar feature vectors from said plurality of second feature vectors (FV-2) based on said first feature vector (FV-1); - means for selecting (SEL-IMG-S) said similar medical image IMG-S from said plurality of second medical images (IMG-2), wherein said similar medical image (IMG-S) is associated with said similar feature vector (FV-S); and - means for providing (PROV-IMG-S) said similar medical image (IMG-S).

14. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to one of claims 1 to 11.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to one of claims 1 to 11.

Images