Generating synthetic medical representations

Abstract

Systems, methods, and computer programs disclosed herein relate to generating synthetic medical representations, such as images.

Description

[0001] Generating synthetic medical representations

[0002] FIELD OF THE DISCLOSURE

[0003] Systems, methods, and computer programs disclosed herein relate to generating synthetic medical representations, such as images.

[0004] BACKGROUND

[0005] Artificial intelligence is increasingly finding its way into medicine. Machine learning models are being used not only to recognize signs of disease in medical images of the human or animal body (see, for example, WO2018202541A1, WO2020229152A1), but also increasingly to generate synthetic (artificial) medical images.

[0006] For example, WO2019074938A1 and WO2022184297A1 describe methods for generating an artificial radiological image showing an examination area of an examination object after application of a standard amount of a contrast agent, although only a smaller amount of contrast agent than the standard amount was applied. The standard amount is the amount recommended by the manufacturer and / or distributor of the contrast agent and / or the amount approved by a regulatory authority and / or the amount listed in a package insert for the contrast agent. The methods described in WO2019074938A1 and WO2022184297A can therefore be used to reduce the amount of contrast agent.

[0007] For example, WO2021052896A1 and WO2021069338A1 describe methods for generating an artificial medical image showing an examination area of an examination object in a first time period. The artificial medical image is generated using a trained machine learning model based on medical images showing the examination area in a second time period. The method can be used, for example, to speed up radiological examinations. Instead of measuring radiological images over a longer period of time, radiological images are measured only within a part of the time period and one or more radiological images are predicted for the remaining part of the time period using the trained model.

[0008] For example, US11170543B2B2 and US11181598B2 describe methods for generating fully-sampled MRI data from under-sampled MRI data using machine learning models.

[0009] For example, WO2016175755A1 and WO2014036473A1 describe methods for generating a high radiation dose CT image based on a low radiation dose CT image using machine learning models.

[0010] The machine learning models disclosed in the cited publications are or include convolutional neural networks. Such machine learning models can be difficult to train, and they often require extensive tuning of hyperparameters; such models can be unstable and sometimes produce images that are not realistic or do not match the training data. Overfitting is a frequently observed problem (see, e.g., P. Thanapol et al. : Reducing Overfitting and Improving Generalization in Training Convolutional Neural Network (CNN) under Limited Sample Sizes in Image Recognition, 2020, 5thInternational Conference on Information Technology (InCIT), pp. 300-305, doi: 10.1109 / InCIT50588.2020.9310787).

[0011] SUMMARY

[0012] These problems are addressed by the subject matter of the independent claims of the present disclosure. Exemplary embodiments are defined in the dependent claims, the description, and the drawings.

[0013] In a first aspect, the present disclosure relates to a computer-implemented method for generating a synthetic medical representation, the method comprising the steps: (a) providing a pre-trained conditional generative model, wherein the pre-trained conditional generative model was trained to reconstruct a plurality of reference representations of an examination area of a plurality of examination objects,

[0014] (b) providing a representation of the examination area of a new examination object,

[0015] (c) providing starting data,

[0016] (d) generating a synthetic representation of the examination area of the new examination object based on the starting data and one or more conditions using the pre -trained conditional generative model,

[0017] (e) generating a transformed synthetic representation based on the synthetic representation,

[0018] (f) quantifying a deviation between the representation of the examination area of the new examination object and the transformed synthetic representation,

[0019] (g) reducing the deviation by modifying the starting data and / or the one or more conditions,

[0020] (h) repeating steps (d) to (g) until a stop criterion is reached,

[0021] (i) outputting and / or storing the synthetic representation of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system.

[0022] In another aspect, the present disclosure provides a computer system comprising: a processing unit; and a memory storing an application program configured to perform, when executed by the processing unit, an operation, the operation comprising:

[0023] (a) providing a pre-trained conditional generative model, wherein the pre-trained conditional generative model was trained to reconstruct a plurality of reference representations of an examination area of a plurality of examination objects,

[0024] (b) providing a representation of the examination area of a new examination object,

[0025] (c) providing starting data,

[0026] (d) generating a synthetic representation of the examination area of the new examination object based on the starting data and one or more conditions using the pre -trained conditional generative model,

[0027] (e) generating a transformed synthetic representation based on the synthetic representation,

[0028] (f) quantifying a deviation between the representation of the examination area of the new examination object and the transformed synthetic representation,

[0029] (g) reducing the deviation by modifying the starting data and / or the one or more conditions,

[0030] (h) repeating steps (d) to (g) until a stop criterion is reached,

[0031] (i) outputting and / or storing the synthetic representation of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system. In another aspect, the present disclosure provides a non-transitory computer readable storage medium having stored thereon software instructions that, when executed by a processing unit of a computer system, cause the computer to perform the following steps:

[0032] (a) providing a pre-trained conditional generative model, wherein the pre-trained conditional generative model was trained to reconstruct a plurality of reference representations of an examination area of a plurality of examination objects,

[0033] (b) providing a representation of the examination area of a new examination object,

[0034] (c) providing starting data,

[0035] (d) generating a synthetic representation of the examination area of the new examination object based on the starting data and one or more conditions using the pre -trained conditional generative model,

[0036] (e) generating a transformed synthetic representation based on the synthetic representation,

[0037] (f) quantifying a deviation between the representation of the examination area of the new examination object and the transformed synthetic representation,

[0038] (g) reducing the deviation by modifying the starting data and / or the one or more conditions,

[0039] (h) repeating steps (d) to (g) until a stop criterion is reached,

[0040] (i) outputting and / or storing the synthetic representation of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system.

[0041] BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Fig. 1 shows schematically by way of example the (pre-)training of a conditional generative model.

[0043] Fig. 2 shows schematically by way of example the use of a pre-trained conditional generative model to generate a synthetic representation of the examination area of a new examination object.

[0044] Fig. 3 shows schematically another example of using a pre-trained conditional generative model to generate a synthetic representation of the examination area of a new examination object.

[0045] Fig. 4 shows schematically another example of using a pre-trained conditional generative model to generate a synthetic representation of the examination area of a new examination object.

[0046] Fig. 5 illustrates a computer system according to some example implementations of the present disclosure in more detail.

[0047] Fig. 6 schematically shows an embodiment of the computer-implemented method for generating a synthetic representation in the form of a flow chart.

[0048] DETAILED DESCRIPTION

[0049] Various example embodiments will be more particularly elucidated below without distinguishing between the aspects of the disclosure (method, computer system, computer-readable storage medium). On the contrary, the following elucidations are intended to apply analogously to all the aspects of the disclosure, irrespective of in which context (method, computer system, computer-readable storage medium) they occur.

[0050] If steps are stated in an order in the present description or in the claims, this does not necessarily mean that the disclosure is restricted to the stated order. On the contrary, it is conceivable that the steps can also be executed in a different order or else in parallel to one another, unless, for example one step builds upon another step, this requiring that the building step be executed subsequently (this being, however, clear in the individual case). The stated orders may thus be exemplary embodiments of the present disclosure.

[0051] As used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” As used in the specification and the claims, the singular form of “a”, “an”, and “the” include plural referents, unless the context clearly dictates otherwise. Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has”, “have”, “having”, or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. Further, the phrase “based on” may mean “in response to” and be indicative of a condition for automatically triggering a specified operation of an electronic device (e.g., a controller, a processor, a computing device, etc.) as appropriately referred to herein.

[0052] Some implementations of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all implementations of the disclosure are shown. Indeed, various implementations of the disclosure may be embodied in many different forms and should not be construed as limited to the implementations set forth herein; rather, these example implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0053] The terms used in this disclosure have the meaning that these terms have in the prior art, in particular in the prior art cited in this disclosure, unless otherwise indicated.

[0054] The present disclosure provides means for generating a synthetic representation of an examination area of an examination object.

[0055] In an embodiment of the present disclosure, the “examination object” is a living being.

[0056] In an embodiment of the present disclosure, the “examination object” is a mammal.

[0057] In an embodiment of the present disclosure, the “examination object” is a human.

[0058] The “examination area” is a part of the examination object, for example an organ or part of an organ or a plurality of organs or another part of the examination object.

[0059] For example, the examination area may be a liver, kidney, heart, lung, brain, stomach, bladder, pancreas, prostate, intestine, thyroid, breast, uterus, skin, eye or a part of said parts or another part of the body of a mammal (for example a human).

[0060] In an embodiment, the examination area includes a liver or part of a liver or the examination area is a liver or part of a liver of a mammal, e.g. a human.

[0061] In a further embodiment, the examination area includes a brain or part of a brain or the examination area is a brain or part of a brain of a mammal, e.g. a human.

[0062] In a further embodiment, the examination area includes a heart or part of a heart or the examination area is a heart or part of a heart of a mammal, e.g. a human.

[0063] In a further embodiment, the examination area includes a thorax or part of a thorax or the examination area is a thorax or part of a thorax of a mammal, e.g. a human.

[0064] In a further embodiment, the examination area includes a stomach or part of a stomach or the examination area is a stomach or part of a stomach of a mammal, e.g. a human. In a further embodiment, the examination area includes a pancreas or part of a pancreas or the examination area is a pancreas or part of a pancreas of a mammal, e.g. a human.

[0065] In a further embodiment, the examination area includes a kidney or part of a kidney or the examination area is a kidney or part of a kidney of a mammal, e.g. a human.

[0066] In a further embodiment, the examination area includes one or both lungs or part of a lung of a mammal, e.g. a human.

[0067] In a further embodiment, the examination area includes a breast or part of a breast or the examination area is a breast or part of a breast of a female mammal, e.g. a female human.

[0068] In a further embodiment, the examination area includes a prostate or part of a prostate or the examination area is a prostate or part of a prostate of a male mammal, e.g. a male human.

[0069] The term “synthetic” means that the representation is not the result of a physical measurement on a real object under examination, but that the representations has been generated (calculated) by a generative machine learning model. A synonym for the term “synthetic” is the term “artificial”.

[0070] The synthetic representation may be an image, i.e., a representation in real space; however, it may also be a representation in another space, such as a representation in frequency space or a representation in projection space.

[0071] In a representation in real space, also referred to in this disclosure as real-space representation, the examination area is normally represented by a number of image elements (for example pixels or voxels or doxels), which may for example be in a raster arrangement in which each image element represents a part of the examination area, wherein each image element may be assigned a colour value or grey value. The colour value or grey value represents a signal intensity, for example the attenuation of X- rays in case of X-ray images. A format widely used in radiology for storing and processing representations in real space is the DICOM format. DICOM (Digital Imaging and Communications in Medicine) is an open standard for storing and exchanging information in medical image data management.

[0072] In a representation in frequency space, also referred to in this description as frequency-space representation, the examination area is represented by a superposition of fundamental vibrations. For example, the examination area may be represented by a sum of sine and cosine functions having different amplitudes, frequencies, and phases. The amplitudes and phases may be plotted as a function of the frequencies, for example, in a two- or three-dimensional plot. Normally, the lowest frequency (origin) is placed in the centre. The further away from this centre, the higher the frequencies. Each frequency can be assigned an amplitude representing the frequency in the frequency-space representation and a phase indicating the extent to which the respective vibration is shifted towards a sine or cosine vibration. The k-space data produced by magnetic resonance imaging (MRI) is an example of a representation in frequency space.

[0073] A representation in real space can for example be converted (transformed) by a Fourier transform into a representation in frequency space. Conversely, a representation in frequency space can for example be converted (transformed) by an inverse Fourier transform into a representation in real space.

[0074] Details about real-space depictions and frequency-space depictions and their respective interconversion are described in numerous publications, see for example https: / / see.stanford.edu / materials / lsoftaee261 / book-fall-07.pdf.

[0075] The representation of an examination area in the projection space can, for example, be the result of a computer tomographic examination prior to image reconstruction. In other words: the raw data obtained in the computed tomography examination can be understood as a projection-space representation. In computed tomography, the intensity or attenuation of X-ray radiation as it passes through the examination object is measured. From this, projection values can be calculated. In a second step, the object information encoded by the projection is transformed into an image (real-space depiction) through a computer-aided reconstruction. The reconstruction can be effected with the Radon transform. The Radon transform describes the link between the unknown examination object and its associated projections.

[0076] Details about the transformation of projection data into a real-space representation are described in numerous publications, see for example K. Catch: The Radon Transformation and Its Application in Tomography, Journal of Physics Conference Series 1903(l):012066.

[0077] For the sake of simplicity, the aspects of present disclosure are described in some parts of this disclosure using the example of images as representations; however, this should not be construed as limiting the disclosure. A person skilled in the art of image processing will know how the teachings of the present disclosure can be applied to representations other than images.

[0078] The term “image” as used herein means a data structure that represents a spatial distribution of a physical signal. The spatial distribution may be of any dimension, for example 2D, 3D, 4D or any higher dimension. The spatial distribution may be of any shape, for example forming a grid and thereby defining pixels or voxels, the grid being possibly irregular or regular. The physical signal may be any signal, for example proton density, tissue echogenicity, tissue radiolucency, measurements related to the blood flow, information of rotating hydrogen nuclei in a magnetic field, color, level of gray, depth, surface or volume occupancy, such that the image may be a 2D or 3D RGB / grayscale / depth image, or a 3D surface / volume occupancy model. An image is usually composed of discrete image elements (e.g., pixels for 2D images, voxels for 3D images, doxels for 4D images).

[0079] In an embodiment of the present disclosure, the synthetic representation is a synthetic medical representation.

[0080] A medical representation is a representation of the human body or a part thereof or a representation of the body of an animal or a part thereof. Medical representations are often used, e.g., for diagnostic and / or treatment purposes.

[0081] Techniques for generating medical representations include X-ray radiography, computerized tomography, fluoroscopy, magnetic resonance imaging, ultrasonography, endoscopy, elastography, tactile imaging, thermography, microscopy, positron emission tomography, optical coherence tomography, fundus photography, and others.

[0082] Examples of medical representations include CT (computer tomography) scans, X-ray images, MRI (magnetic resonance imaging) scans, fluorescein angiography images, OCT (optical coherence tomography) scans, histological images, ultrasound images, fundus images and / or others.

[0083] In an embodiment, the medical representation is a microscopic image, such as a whole slide histological image of a tissue of a human body. The histological image can be an image of a stained tissue sample. One or more dyes can be used to create the stained image. Usual dyes are hematoxylin and eosin.

[0084] In another embodiment, the medical representation is a radiological representation. “Radiology” is the branch of medicine concerned with the application of electromagnetic radiation and mechanical waves (including, for example, ultrasound diagnostics) for diagnostic, therapeutic and / or scientific purposes. In addition to X-rays, other ionizing radiation such as gamma rays or electrons are also used. Since a primary purpose is imaging, other imaging procedures such as sonography and magnetic resonance imaging (MRI) are also included in radiology, although no ionizing radiation is used in these procedures. Thus, the term “radiology” as used in the present disclosure includes, in particular, the following examination procedures: computed tomography, magnetic resonance imaging, sonography, positron emission tomography (PET).

[0085] The radiological representation can be, e.g., a 2D or 3D CT scan or MRI scan. The radiological representation may be a representation with a contrast agent or without a contrast agent. It may also be multiple representations, one or more of which were generated using a contrast agent and one or more of which were generated without a contrast agent. In an embodiment of the present disclosure, the synthetic medical representation is a synthetic MRI representation obtained from under-sampled k-space data.

[0086] In another embodiment of the present disclosure, the synthetic medical representation is a synthetic medical representation of an examination area of an examination object after administration of a second amount of a contrast agent based on a measured representation of the examination area of the examination object after administration of a first amount of the contrast agent, wherein the first amount and the second amount are different from each other.

[0087] In another embodiment of the present disclosure, the synthetic medical representation is a synthetic medical representation of an examination area of an examination object as a result of a radiological examination with a second radiation dose, based on a measured representation of the examination area of the examination object as a result of a radiological examination with a first radiation dose, wherein the first radiation dose and the second radiation dose are different from each other.

[0088] In another embodiment of the present disclosure, the synthetic medical representation is a synthetic medical representation of an examination area of an examination object at a second time point before or after the application of a contrast agent, based on a measured representation of the examination area of the examination object at a first time point before or after the application of the contrast agent, wherein the first time point and the second time point are different from each other.

[0089] Synthetic medical representations are generated using a pre-trained conditional generative model.

[0090] A “generative model” is a type of machine learning model that is designed to learn and generate new data that resembles the training data it was trained on. Generative models capture the underlying distribution of the training data and can generate samples from that distribution.

[0091] A “conditional generative model” is a type of generative model that generates data (in this case, synthetic representations) given certain conditions or constraints. Conditional generative models take additional input in the form of a condition that guides the process of generating the synthetic representation. In general, this condition can be anything that provides some sort of context for the generation process, such as a class label, a text description, an image, or any other piece of information.

[0092] A “machine learning model” may be understood as a computer implemented data processing architecture. The machine learning model can receive input data and provide output data based on that input data and on parameters of the machine learning model (model parameters). The machine learning model can learn a relation between input data and output data through training. In training, parameters of the machine learning model may be adjusted in order to provide a desired output for a given input.

[0093] The process of training a machine learning model involves providing a machine learning algorithm (that is the learning algorithm) with training data to learn from. The term “trained machine learning model” refers to the model artifact that is created by the training process. The training data usually contain the correct answer, which is referred to as the target. The learning algorithm finds patterns in the training data that map input data to the target, and it outputs a trained machine learning model that captures these patterns.

[0094] In the training process, training data are inputted into the machine learning model and the machine learning model generates an output. The output is compared with the (known) target. Parameters of the machine learning model are modified in order to reduce the deviations between the output and the (known) target to a (defined) minimum.

[0095] In general, a loss function can be used for training, where the loss function can quantify the deviations between the output and the target. The loss function may be chosen in such a way that it rewards a wanted relation between output and target and / or penalizes an unwanted relation between an output and a target. Such a relation can be, e.g., a similarity, or a dissimilarity, or another relation.

[0096] A loss function can be used to calculate a loss for a given pair of output and target. The aim of the training process can be to modify (adjust) parameters of the machine learning model in order to reduce the loss to a (defined) minimum. In the case of the present disclosure, a conditional generative model was pre-trained to reconstruct a plurality of reference representations of an examination area of a plurality of examination objects.

[0097] The term “plurality” means more than ten, e.g. more than one hundred.

[0098] Each examination object of the plurality of examination objects is a living being such as an animal, e.g. a mammal, and / or a human. In an embodiment of the present disclosure, each examination object is a human.

[0099] The examination area is a part of the examination object, e.g., a liver, kidney, heart, lung, brain, stomach, bladder, prostate, intestine, eye, thyroid, pancreas, breast, uterus, skin, or a part of said parts or another part of the examination object.

[0100] The examination area is usually the same for all examination objects.

[0101] The term “reference” is used in this description to distinguish the phase of pre-training the conditional generative model from the phase of using the pre-trained model for generating a synthetic representation. The term “reference” otherwise has no limitation on meaning. A “reference representation” is a representation of the examination area of an examination object which is used to pre-train the conditional generative model. If the pre-trained conditional generative model is used to generate a synthetic representation for an examination object, the examination object is also referred to as a “new” examination object in this disclosure. Again, the term “new” is used only to distinguish the pre-training phase from the phase of using the pre -trained conditional generative model for generating synthetic representations.

[0102] The pre-training phase includes the following steps, which are explained in more detail below: providing a training data set, the training data set comprising a plurality of reference representations of an examination area of a plurality of examination objects, providing a conditional generative model, pre-training the conditional generative model on the training data set, storing the pre-trained conditional generative model.

[0103] Each reference representation of the examination area of each examination object may be a medical image or a representation from which a medical image can be generated or a representation that can be converted into a medical image.

[0104] Each reference representation of the examination area of each examination object can be a representation in real space (image space, spatial domain), a representation in frequency space (frequency domain), a representation in the projection space or a representation in another space.

[0105] The reference representations of the training data set can be or comprise annotated representations. In other words, for some or all reference representations, an annotation may be available.

[0106] The annotation may provide information about the nature of the reference representations (e.g., CT representation, MRI representation, microscopic image, and / or the like), what is represented by the reference representation (e.g., which examination area) and / or from which examination object the reference representation was obtained, and / or how the reference representation was created (e.g., which measurement protocol was used, whether a contrast agent was administered, which contrast agent was administered, and / or the amount of contrast agent administered, if applicable, which radiation dose was used, at which time the reference representation was generated (e.g., before or after administration of a contrast agent)), and / or other / further information about the reference representation and / or the content of the reference representation and / or the generation of the reference representation.

[0107] However, it is also possible that the reference representations of the training data set are or comprise non-annotated representations. In an embodiment of the present disclosure, at least some of the reference representations are annotated representations. The training data set is used to pre-train the conditional generative model.

[0108] In an embodiment of the present disclosure, the conditional generative model is or comprises a diffusion model.

[0109] Diffusion models focus on modeling the step-by-step evolution of data distribution from a simple starting point to a more complex distribution. The underlying concept of diffusion models is to transform a simple and easily sampleable distribution, typically a Gaussian distribution, into a more complex data distribution of interest. This transformation is achieved through a series of invertible operations. Once the model learns the transformation process, it can generate new samples by starting from a point in the simple distribution and gradually “diffusing” it to the desired complex data distribution.

[0110] A diffusion model usually comprises a noising model and a denoising model.

[0111] The noising model usually comprises a plurality of noising stages. The noising model is configured to receive input data (e.g., a representation of an examination area of an examination object) and produce noise data in response to receipt of the input data. The noising model introduces noise to the input data to obfuscate the input data after a number of stages, or “timesteps”. The noising model can be or can include a finite number of steps Tor an infinite number of steps (T- co).

[0112] The denoising model is configured to reconstruct the input data from the noise data. The denoising model is configured to produce samples matching the input data after a number of stages.

[0113] For example, the diffusion model may include Markov chains at the noising model and / or denoising model. The diffusion model may be implemented in discrete time, e.g., where each layer corresponds to a timestep. The diffusion model may also be implemented in arbitrarily deep (e.g., continuous) time.

[0114] For example, the diffusion model may be fully Gaussian such that an unbiased estimate of the objective function can be obtained from a single layer. It is thus possible to avoid computing intermediate layers.

[0115] Diffusion models can be conceptually similar to a variational autoencoder (VAE) whose structure and loss function provides for efficient training of arbitrarily deep (e.g., infinitely deep) models. The diffusion model can be trained using variational inference.

[0116] The model can be a latent diffusion model. In such a model, the diffusion approach in case of real-space representations is not performed in pixel space, but in so-called latent space based on a representation of the image, usually a compressed representation (see, e.g., R. Rombach etal. : High-Resolution Image Synthesis with Latent Diffusion Models , arXiv:2112.10752v2).

[0117] The diffusion model may be a Denoising Diffusion Probabilistic Model (DDPM). DDPMs are a class of generative models that work by iteratively adding noise to input data and then learning to denoise from the noisy signal to generate new samples (see, e.g., J. Ho et al. : Denoising Diffusion Probabilistic Models, arXiv:2006. 11239v2).

[0118] The diffusion model may be a Denoising Diffusion Implicit Model (DDIM). A critical drawback of DDPMs is that they require many iterations to produce a high-quality sample. For DDPMs, this is because the generative process (from noise to data) approximates the reverse of the forward diffusion process (from data to noise), which could have thousands of steps; iterating over all the steps is required to produce a single sample. DDIMs are implicit probabilistic models that are closely related to DDPMs, in the sense that they are trained with the same objective function. DDIMs allow for much faster sampling while keeping an equivalent training objective. They do this by estimating the addition of multiple Markov chain steps and adding them all at once. DDIMs construct a class of non-Markovian diffusion processes which makes sampling from reverse process much faster (see, e.g.: J. Song et al. -. Denoising Diffusion Implicit Models , arXiv:2010.02502v4). This modification in the forward process preserves the goal of DDPM and allows for deterministically encoding an image to the noise map.

[0119] Unlike DDPMs, DDIMs enable control over image synthesis owing to the latent space flexibility (attribute manipulation) (see, e.g., K. Preechakul et al:. Diffusion autoencoders: Toward a meaningful and decodable representation, arXiv:2111.15640v3). With DDIM, it is possible to run the generative process backward deterministically to obtain the noise map xT, which represents the latent variable or encoding of a given image x0. In this context, DDIM can be thought of as an image decoder that decodes the latent code XT back to the input image. This process can yield a very accurate reconstruction; however, xj- still does not contain high-level semantics as would be expected from a meaningful representation.

[0120] In a conditional generative model, one or more conditions can be used for denoising starting data and reconstructing an input representation. In general, such a condition can be based on a text or it can be a segmentation map or other information.

[0121] For example, the annotation of each reference representation of the training data set can be used as a condition in reconstructing the reference representation from starting data, e.g. noise data.

[0122] From each annotation, a semantic representation may be generated. In case of text, such a semantic annotation representation can be generated by a text encoder, e.g., a text encoder of a CLIP model (see, e.g., A. Radford et al. '. Learning Transferable Visual Models From Natural Language Supervision, arXiv:2103.00020vl), or any other word embedding technique (see, e.g., S. Selva Birunda and R. Kanniga Devi: A Review on Word Embedding Techniques for Text Classification in Innovatove Data Communication Technologies and Application, Proceedings of ICIDCA 2020, Vol. 59, Springer, ISSN 2367-4512, 267-281; M. T. Pilehvar, J. Camacho-Collados: Embeddings in Natural Language Processing, Morgan&Claypool Publishers 2021, ISBN: 9781636390222).

[0123] If the annotation is not a text fde, another encoder can be used to generate a semantic representation of the annotation, such as an image encoder, an encoder of a pre-trained autoencoder, and / or the like.

[0124] It is for example possible to create a semantic (e.g., compressed) representation of the reference representation of the examination area and use this as a (further) condition (see, e.g., X. Xu et al. : ViT- DAE: Transformer-driven Diffusion Autoencoder for Histopathology Image Analysis, arXiv:2304.01053vl).

[0125] The semantic representation may be used as conditional input to the conditional generative model.

[0126] The purpose of pre-training the conditional generative model is for the conditional generative model to learn what representations of the examination area “look like”, e.g., which structures and / or patterns occur and / or which short-, medium- and / or long-term correlations exist.

[0127] (Pre-)training of conditional generative models is described in detail in numerous scientific publications (see, e.g., H. Walter et al:. Brain Imaging Generation with Latent Diffusion Models, arXiv:2209.07162vl).

[0128] Once the conditional generative model is pre-trained, it can be used to generate a synthetic representation of the examination area of a new examination object. The term “new” may mean that no reference representations of the examination area of the new examination object have been used for the pretraining of the diffusion model.

[0129] In a first step, a representation of the examination area of the new examination object is provided. This representation serves as target data.

[0130] This representation is usually measured data, i.e., the result of a measurement on the new examination object. It is possible that the measured data is processed or transformed to generate the representation of the examination area of the new examination object. For example, the measured data may be k-space data that is subjected to an inverse Fourier transform to generate a real-space representation of the examination area of the new examination object (however, this does not mean that the representation of the examination area of the examination object cannot consist of k-space data).

[0131] As is usual with conditional generative models, the synthetic representation is generated based on starting data, e.g. noise data. Therefore, starting data (e.g., initial noise data) is provided, which forms the starting point for generating the synthetic representation. The starting data can be random Gaussian noise, for example. The conditional generative model may be configured and pre-trained to gradually denoise the initial noise data and generate a synthetic representation of the examination area.

[0132] If one or more conditions were used in the pre-training of the conditional generative model, one or more conditions can also be used in the generation of the synthetic representation.

[0133] In a next step, the synthetic representation is transformed. The result of the transformation operation is a transformed synthetic representation of the examination area of the new examination object.

[0134] The aim of the transformation is to convert the synthetic representation so that it corresponds to the (e.g., measured) representation of the examination area of the new examination object (target data).

[0135] The transformed synthetic representation is compared with the representation of the examination area of the new examination object (target data)

[0136] A loss function may be used to quantify a deviation between the representation of the examination area of the new examination object and the transformed synthetic representation.

[0137] The deviation can be reduced in an optimization process by modifying the starting data and / or the condition(s). A new synthetic representation is generated with the modified starting data and / or the modified condition(s). This is then converted into a transformed synthetic representation and a deviation between the new transformed synthetic representation and the representation of the examination area of the new examination object is quantified. The procedure is continued until a stop criterion is reached.

[0138] Such stop criteria can be for example: a predefined maximum number of steps has been performed, deviations between output data and target data can no longer be reduced by modifying the starting data and / or the condition(s), a predefined minimum of the loss function is reached, and / or an extreme value (e.g., maximum or minimum) of another performance value is reached.

[0139] Once the stop criterion has been reached, the last synthetic representation generated can be output (e.g., shown on a display and / or printed out on a printer), stored in a data memory and / or transferred to another computer system.

[0140] It is also possible that the last generated synthetic representation is first converted into another representation, which is then output, stored and / or transferred to a separate computer system. For example, it is possible for the synthetic representation to be a representation in frequency space or projection space and for this to be converted into a real-space representation.

[0141] As described, the synthetic representation generated by the conditional generative model is transformed and the transformed synthetic representation is compared with the target data in order to quantify a deviation. The transformation of the synthetic representation is a process that reverses what the conditional generative model is intended to achieve. This is explained in more detail in the following examples, without wishing to limit the disclosure to these examples.

[0142] In an embodiment of the present disclosure, the conditional generative model is used to generate a representation of the examination area after the application of a second amount of a contrast agent, based on a representation of the examination area after the application of a first amount of the contrast agent. The first amount is different from the second amount. The first amount may be less than the second amount, or the first amount may be greater than the second amount. If the first amount is less than the second amount, the method of the present disclosure may be used to reduce the amount of contrast agent in a radiologic examination. For example, a first representation of the examination area of an examination object may be generated after application of an amount less than the standard amount, and based on this measured representation, a second representation of the examination area of the examination object after application of the standard amount may be generated using the pre-trained diffusion model. The standard amount is the amount recommended by the manufacturer and / or distributor of the contrast agent and / or the amount approved by a regulatory authority and / or the amount listed in a package insert for the contrast agent.

[0143] Based on starting data (e.g., noise data), the pre-trained conditional generative model should therefore generate a synthetic representation of the examination area after the application of a second amount of contrast agent. The synthetic representation generated by the pre-trained conditional generative model is converted into a transformed synthetic representation, which should correspond to a representation of the examination area of the examination object after the application of a first amount of contrast agent. A measured representation of the examination area of the examination object after the application of the first amount of contrast agent may serve as target data. The starting data and / or one or more conditions may be modified (e.g., in an optimization process) so that the transformed synthetic representation comes as close as possible to the target data.

[0144] A transformation function is therefore required that can convert a synthetic representation of an examination area after the application of a second amount of a contrast agent into a representation of the examination area after the application of a first amount of the contrast agent. Various approaches are described in the literature, some of which are briefly outlined hereinafter.

[0145] In contrast-enhanced MRI images, over a wide range the signal intensity is linearly dependent on the amount of contrast agent used. It is therefore possible to subtract a native image from the synthetic image. Such a native image shows the same examination area without contrast agent. Such a native image is usually generated in MRI examinations before the contrast agent is administered. The result of the subtraction is a representation of the examination area that reflects the contrast agent distribution in the examination area and in which the observed signal intensities are only caused by the contrast agent. This difference representation can be multiplied by a factor a in order to either amplify (a> 1 ) or attenuate (a<l) the signal intensities caused by the contrast agent. The result of the multiplication can then be added to the native representation. The result of the addition is a representation that should match the target data. Alternatively, the result of the multiplication can also be subtracted from the synthetic representation or added to the synthetic representation - depending on whether the transformation is to lead to an attenuation of the contrast agent signal or to an amplification of the contrast agent signal. Such a procedure is disclosed, for example, in: A. Fringuello Mingo et al. : Amplifying the Effects of Contrast Agents on Magnetic Resonance Images Using a Deep Learning Method Trained on Synthetic Data, Investigative Radiology 58(12), 2023, 853-864.

[0146] An analogous approach can also be followed for other contrast-enhanced radiologic methods. If there is no linear relationship between the amount of contrast agent and the signal intensity, the signal intensities caused by contrast agent can also be attenuated or amplified on the basis of another functional relationship; only the functional relationship must be determined empirically.

[0147] In computed tomography (CT), for example, dual-energy CT can be used to generate a contrast map, which can also be amplified or attenuated (see e.g. : D. Grob et al. : Iodine Maps from Subtraction CT or Dual-Energy CT to Detect Pulmonary Emboli with CT Angiography: A Multiple-Observer Study, Radiology 2019, 292: 197-205) and added to another CT representation or subtracted therefrom.

[0148] In another embodiment of the present disclosure, the diffusion model is used to generate synthetic high- dose CT data of an examination area of an examination object using low-dose CT data as target data.

[0149] The terms “high-dose” and “low-dose” refer to the radiation dose and have the definitions commonly used in the state of the art (see, e.g., WO2016 / 175755A1). The terms “high-dose” and “low-dose” refer to the respective other dose. This means that the low dose is a lower dose than the high dose and the high dose is a higher dose than the low dose.

[0150] CT data may be a CT image (e.g., a CT scan) or another CT representation (e.g., a representation of the examination area of the examination object in projection space). In a first step, low-dose CT data of an examination area of a new examination object is received as target data.

[0151] Based on noise data, the pre-trained diffusion model generates synthetic high-dose CT data of the examination area of the examination object. A transformation is used to convert the synthetic high-dose CT data into synthetic low-dose CT data. The synthetic low-dose CT data is compared with the target data: deviations are quantified. The starting data and / or one or more conditions may be modified (e.g. in an optimization process) in order to reduce the deviations.

[0152] For this embodiment, a transformation operation is required that transforms high-dose CT data into low- dose CT data. This can be done, for example, in case of CT images, by reducing image contrast of the high-dose CT image and adding noise to the high-dose CT image (see, e.g., L.W. Goldmann: Principles of CT: Radiation Dose and Image Quality, Journal of Nuclear Medicine Technology, 2007, 35 (4) 213- 225).

[0153] In another embodiment of the present disclosure, the conditional generative model is used to generate fully-sampled MRI data based on under-sampled MRI data. The terms “fully-sampled” and “undersampled” (also referred to as “sub-sampled”) have the definitions commonly used in the prior art (see, e.g., US11170543B2B2). A “fully-sampled” acquisition refers to the process of collecting magnetic resonance data from the entire k-space with a sampling density that meets or exceeds the Nyquist criterion. This criterion ensures that the spatial frequencies are sampled sufficiently to accurately reconstruct the image without aliasing artifacts. A fully-sampled dataset provides the maximum spatial resolution and signal-to-noise ratio achievable under the given scanning parameters and constraints, allowing for the generation of high-quality images that accurately represent the anatomy or pathology of interest. In contrast, an “under-sampled” acquisition in magnetic resonance imaging denotes the process of collecting magnetic resonance data from k-space at a sampling density below the Nyquist criterion. This results in a reduced number of measurements compared to a fully-sampled approach, leading to faster data acquisition times but at the cost of potential image artifacts such as aliasing.

[0154] In a first step, under-sampled MRI data (e.g., under-sampled k-space data or an MRI image generated therefrom) of an examination area of a new examination object is received as target data.

[0155] Based on starting data (e.g., noise data), the pre-trained diffusion model generates synthetic fully sampled MRI data of the examination area of the examination object. A transformation is used to convert the synthetic fully-sampled MRI data into synthetic under-sampled MRI data. The synthetic undersampled MRI data is compared with the target data: deviations are quantified. In an optimization procedure, the starting data and / or one or more conditions are modified (optimized) in order to reduce the deviations.

[0156] For this embodiment, a transformation operation is required that transforms fully-sampled MRI data into under-sampled MRI data. This can be done, for example, in case of k-space data, by eliminating some frequencies from the fully-sampled MRI data (see, e.g., A. Deshmane et al. : Parallel MR Imaging, J Magn Reson Imaging, 2012, 36(1): 55-72).

[0157] There are numerous other ways in which the pre -trained conditional generative model can be used, such as changing the resolution of a representation, segmentation, predicting a state based on a previous or subsequent state, and much more.

[0158] The aspects of the present disclosure are explained in more detail below with reference to examples and drawings, without the intention to limit the disclosure to the examples or the features and combinations of features shown in the drawings.

[0159] Fig. 1 shows schematically by way of example the (pre-)training of a conditional generative model of the present disclosure.

[0160] The conditional generative model (CGM) is a diffusion model. The conditional generative model (CGM) comprises a noising model (NM) and a denoising model (DM). The conditional generative model (CGM) is (pre-)trained on training data (TD).

[0161] The training data (TD) comprise a multitude of reference representations (Rl, R2, R3, R4, R5, R6) of an examination area of a multitude of examination objects.

[0162] In the example shown in Fig. 1, the examination area comprises the brain of a multitude of humans.

[0163] Some reference representations (Rl, R4, R5, R6) are annotated, i.e., for some reference representations (Rl, R4, R5, R6) there is an annotation (Al, A4, A5, A6) that can indicate what the reference representation represents and / or how it was generated and / or may include other / further information.

[0164] The conditional generative model (CGM) may be configured and trained to gradually add noise to the reference representations (Rl, R2, R3, R4, R5, R6) and regenerate the reference representations from the noise data (denoising). The annotations can be used as conditions. As shown in Fig. 1, a numerical representation (e.g., a vector) can be generated from the annotation A6 with the help of a text encoder (TE), which is used as a condition in the reconstruction of the reference representation R6. If the annotation is not a text file, another encoder can be used to generate a semantic representation of the annotation, such as an image encoder, an encoder of a pre-trained autoencoder, and / or the like.

[0165] In Fig. 1, the reconstructed reference representations are marked with the reference signs R1R, R2R, R3R, R4R, R5R, and R6R.

[0166] Of course, augmentation and masking techniques can be used in the reconstruction of the reference representations.

[0167] Once the conditional generative model (CGM) has been pre-trained, it can be used to generate a synthetic representation of the examination area of a new examination object. This is shown in Figs. 2, 3 and 4.

[0168] Fig. 2 shows schematically by way of example the use of a pre-trained conditional generative model model to generate a synthetic representation of the examination area of a new examination object.

[0169] The pre-trained conditional generative model (CGM‘) comprises a trained denoising model (DM‘). The noising model NM shown in Fig. 1 is not required to generate synthetic representations.

[0170] In a first step, a representation (R) of the examination area of a new examination object is received or provided.

[0171] Since the pre-trained conditional generative model (CGM1) has been pre-trained to reconstruct representations of human brains (see Fig. 1), the representation R also represents a human brain. The representation R is annotated, i.e., there is an annotation A that can indicate what the representation R represents and / or how it was generated and / or may include other / further information.

[0172] The representation R serves as the target data. A numerical representation is generated from the annotation A with the help of the text encoder TE, which is used as a condition for generating the synthetic representation SR. If the annotation (A) is not a text file, another encoder can be used to generate a semantic representation of the annotation, such as an image encoder, an encoder of a pretrained autoencoder, and / or the like.

[0173] The synthetic representation (SR) is generated based on the starting data (SD). The pre-trained conditional generative model (CGM1) is configured and trained to generate the synthetic representation (SR) based on the starting data (SD) under the conditions specified by the annotation (A). In a further step, the synthetic representation (SR) is converted into a transformed synthetic representation (SRT). The transformed synthetic representation (SRT) is compared with the representation (R). Deviations between the transformed synthetic representation (SRT) and the representation (R) are quantified (e.g., by using a loss function (LF)) and the deviations are reduced by modifying the starting data (SD) and / or one or more conditions specified by the annotation (A) in an optimization process. Modifying one or more conditions may also include modifying the representation of the annotation. Once a stop criterion has been reached, the last synthetic representation generated can be output (e.g., shown on a display and / or printed out on a printer), stored in a data memory and / or transferred to another computer system.

[0174] Such stop criteria can be for example: a predefined maximum number of steps has been performed, deviations between the transformed synthetic representation (SRT) and the representation (R) can no longer be reduced by modifying the starting data (SD) and / or one or more conditions specified by the annotation (A), a predefined minimum of the loss function is reached, and / or an extreme value (e.g., maximum or minimum) of another performance value is reached.

[0175] Fig. 3 shows schematically another example of using a pre-trained conditional generative model to generate a synthetic representation of the examination area of a new examination object.

[0176] In a first step, a low-dose CT representation (R) is received. The low-dose CT representation (R) may be one CT representation of a set of dual-energy CT representations. The low-dose CT representation (R) represents the examination area of an examination object. The examination area comprises a lung of a human being. A first radiation dose was used to generate the low-dose CT representation (R).

[0177] The low-dose CT representation (R) is annotated, i.e., there is an annotation (A) which may indicate what the low-dose CT representation represents (R) and / or how it was generated.

[0178] A text encoder (TE) is used to generate a numeric representation based on the annotation (A), which is used as a condition for generating the synthetic representation (SR). If the annotation (A) is not a text file, another encoder can be used to generate a semantic representation of the annotation, such as an image encoder, an encoder of a pre-trained autoencoder, and / or the like.

[0179] The pre-trained conditional generative model (CGM‘) comprises a pre-trained denoising model (DM‘). The pre-trained conditional generative model (CGM‘) may have been trained in a process as described with reference to Fig. 1. In an embodiment of the present disclosure, the pe-trained conditional generative model (CGM‘) was trained based on representations of human lungs, as the pre-trained conditional generative model (CGM‘) is used in the example shown in Fig. 3 to generate a synthetic representation (SR) of a human lung.

[0180] The pre-trained conditional generative model (CGM‘) is configured and trained to generate a synthetic representation (SR) of the examination area of the examination object based on starting data (SD). The synthetic representation (SR) is a synthetic high-dose CT representation.

[0181] The synthetic representation (SR) shows the examination area of the examination object as it would look if a second radiation dose had been used. The second radiation dose is higher than the first radiation dose.

[0182] The synthetic representation (SR) is converted into a transformed synthetic representation (SRT). The transformed synthetic representation (SRT) is a synthetic low-dose CT representation; it shows the examination area of the examination object as it would look if the first radiation dose had been used.

[0183] A loss function (LF) is used to quantify deviations between the transformed synthetic representation (SRT) and the low-dose CT representation (R). In an optimization process, the starting data (SD) and / or one or more conditions specified by the annotation (A) are modified to reduce the deviations.

[0184] Once a stop criterion has been reached, the last synthetic representation generated can be output (e.g., shown on a display and / or printed out on a printer), stored in a data memory and / or transferred to another computer system.

[0185] Fig. 4 shows schematically another example of using a pre-trained machine learning model to generate a synthetic representation of the examination area of a new examination object.

[0186] In a first step, a first representation (NR) and a second representation (R) are received. The first representation (NR) and the second representation (R) are radiologic representations, such as CT representations or MRI representations. The first representation (NR) is a native representation; it represents an examination area of a new examination object without contrast agent. The second representation (R) represents the same examination area of the same examination object after the application of a first amount of contrast agent. The examination area comprises the lungs of a human. At least the second representation (R) is annotated, i.e., there is an annotation (A) which may indicate what the second representation comprises and / or how it was generated and / or may include other / fiirther information. The annotation (A) may also indicate whether a contrast agent was used, and / or which contrast agent was used, and / or what amount of the contrast agent was used.

[0187] A text encoder (TE) is used to generate a numeric representation based on the annotation (A), which is used as a condition for generating the synthetic representation (SR). If the annotation (A) is not a text file, another encoder can be used to generate a semantic representation of the annotation, such as an image encoder, an encoder of a pre-trained autoencoder, and / or the like.

[0188] The pre-trained conditional generative model (CGM‘) comprises a pre-trained denoising model (DM1). The pre-trained conditional generative model (CGM‘) may have been trained in a process as described with reference to Fig. 1. In an embodiment of the present disclosure, the conditional generative model (CGM‘) was trained based on representations of human lungs, as the pre-trained conditional generative model (CGM‘) is used in the example shown in Fig. 4 to generate a synthetic representation (SR) of a human lung.

[0189] The pre-trained conditional generative model (CGM‘) is configured and trained to generate a synthetic representation (SR) of the examination area of the examination object based on starting data (SD). The synthetic representation (SR) represents the examination area of the examination object after application of a second amount of a contrast agent, where in the example shown in Fig. 4, the second amount is larger than the first amount.

[0190] The synthetic representation (SR) is converted into a transformed synthetic representation (SRT). The first representation (NR) is also used for the transformation. The transformed synthetic representation (SRT) represents the examination area of the examination object after application of the first amount of a contrast agent.

[0191] For obtaining the transformed synthetic representation (SRT) it is possible, for example, to subtract the first representation (NR) from the synthetic representation (SR), multiply the result of the subtraction by a factor (e.g., a factor that is greater than 1, or a factor that is less than 1) and either add the result of the multiplication to the first representation (NR) or subtract it from the synthetic representation (SR) to obtain the transformed synthetic representation (SRT). Other transformations are also possible.

[0192] A loss function (LF) is used to quantify deviations between the transformed synthetic representation (SRT) and the second representation (R). In an optimization process, the starting data (ND) and / or one or more conditions specified by the annotation (A) are modified to reduce the deviations.

[0193] Once a stop criterion has been reached, the last synthetic representation generated can be output (e.g., shown on a display and / or printed out on a printer), stored in a data memory and / or transferred to another computer system.

[0194] The operations in accordance with the teachings herein may be performed by at least one computer specially constructed for the desired purposes or general purpose computer specially configured for the desired purpose by at least one computer program stored in a typically non-transitory computer readable storage medium.

[0195] The term “non-transitory” is used herein to exclude transitory, propagating signals or waves, but to otherwise include any volatile or non-volatile computer memory technology suitable to the application.

[0196] The term “computer” should be broadly construed to cover any kind of electronic device with data processing capabilities, including, by way of non-limiting example, personal computers, servers, embedded cores, computing system, communication devices, processors (e.g., digital signal processor (DSP)), microcontrollers, field programmable gate array (FPGA), application specific integrated circuit (ASIC), etc.) and other electronic computing devices.

[0197] The term “process” as used above is intended to include any type of computation or manipulation or transformation of data represented as physical, e.g., electronic, phenomena which may occur or reside e.g., within registers and / or memories of at least one computer or processor. The term processor includes a single processing unit or a plurality of distributed or remote such units.

[0198] Fig. 5 illustrates a computer system (1) according to some example implementations of the present disclosure in more detail. The computer may include one or more of each of a number of components such as, for example, processing unit (20) connected to a memory (50) (e.g., storage device).

[0199] The processing unit (20) may be composed of one or more processors alone or in combination with one or more memories. The processing unit is generally any piece of computer hardware that is capable of processing information such as, for example, data, computer programs and / or other suitable electronic information. The processing unit is composed of a collection of electronic circuits some of which may be packaged as an integrated circuit or multiple interconnected integrated circuits (an integrated circuit at times more commonly referred to as a “chip”). The processing unit may be configured to execute computer programs, which may be stored onboard the processing unit or otherwise stored in the memory (50) of the same or another computer.

[0200] The processing unit (20) may be a number of processors, a multi-core processor or some other type of processor, depending on the particular implementation. Further, the processing unit may be implemented using a number of heterogeneous processor systems in which a main processor is present with one or more secondary processors on a single chip. As another illustrative example, the processing unit may be a symmetric multi -processor system containing multiple processors of the same type. In yet another example, the processing unit may be embodied as or otherwise include one or more ASICs, FPGAs or the like. Thus, although the processing unit may be capable of executing a computer program to perform one or more functions, the processing unit of various examples may be capable of performing one or more functions without the aid of a computer program. In either instance, the processing unit may be appropriately programmed to perform functions or operations according to example implementations of the present disclosure.

[0201] The memory (50) is generally any piece of computer hardware that is capable of storing information such as, for example, data, computer programs (e.g., computer-readable program code (60)) and / or other suitable information either on a temporary basis and / or a permanent basis. The memory may include volatile and / or non-volatile memory, and may be fixed or removable. Examples of suitable memory include random access memory (RAM), read-only memory (ROM), a hard drive, a flash memory, a thumb drive, a removable computer diskette, an optical disk, a magnetic tape or some combination of the above. Optical disks may include compact disk - read only memory (CD-ROM), compact disk - read / write (CD-R / W), DVD, Blu-ray disk or the like. In various instances, the memory may be referred to as a computer-readable storage medium. The computer-readable storage medium is a non-transitory device capable of storing information, and is distinguishable from computer-readable transmission media such as electronic transitory signals capable of carrying information from one location to another. Computer-readable medium as described herein may generally refer to a computer-readable storage medium or computer-readable transmission medium.

[0202] In addition to the memory (50), the processing unit (20) may also be connected to one or more interfaces for displaying, transmitting and / or receiving information. The interfaces may include one or more communications interfaces and / or one or more user interfaces. The communications interface(s) may be configured to transmit and / or receive information, such as to and / or from other computer(s), network(s), database(s) or the like. The communications interface may be configured to transmit and / or receive information by physical (wired) and / or wireless communications links. The communications interface(s) may include interface(s) (41) to connect to a network, such as using technologies such as cellular telephone, Wi-Fi, satellite, cable, digital subscriber line (DSL), fiber optics and the like. In some examples, the communications interface(s) may include one or more short-range communications interfaces (42) configured to connect devices using short-range communications technologies such as NFC, RFID, Bluetooth, Bluetooth LE, ZigBee, infrared (e.g., IrDA) or the like.

[0203] The user interfaces may include a display (30). The display may be configured to present or otherwise display information to a user, suitable examples of which include a liquid crystal display (LCD), lightemitting diode display (LED), plasma display panel (PDP) or the like. The user input interface(s) (11) may be wired or wireless, and may be configured to receive information from a user into the computer system (1), such as for processing, storage and / or display. Suitable examples of user input interfaces include a microphone, image or video capture device, keyboard or keypad, joystick, touch-sensitive surface (separate from or integrated into a touchscreen) or the like. In some examples, the user interfaces may include automatic identification and data capture (AIDC) technology (12) for machine-readable information. This may include barcode, radio frequency identification (RFID), magnetic stripes, optical character recognition (OCR), integrated circuit card (ICC), and the like. The user interfaces may further include one or more interfaces for communicating with peripherals such as printers and the like.

[0204] As indicated above, program code instructions may be stored in memory, and executed by processing unit that is thereby programmed, to implement functions of the systems, subsystems, tools and their respective elements described herein. As will be appreciated, any suitable program code instructions may be loaded onto a computer or other programmable apparatus from a computer-readable storage medium to produce a particular machine, such that the particular machine becomes a means for implementing the functions specified herein. These program code instructions may also be stored in a computer-readable storage medium that can direct a computer, processing unit or other programmable apparatus to function in a particular manner to thereby generate a particular machine or particular article of manufacture. The instructions stored in the computer-readable storage medium may produce an article of manufacture, where the article of manufacture becomes a means for implementing functions described herein. The program code instructions may be retrieved from a computer-readable storage medium and loaded into a computer, processing unit or other programmable apparatus to configure the computer, processing unit or other programmable apparatus to execute operations to be performed on or by the computer, processing unit or other programmable apparatus.

[0205] Retrieval, loading and execution of the program code instructions may be performed sequentially such that one instruction is retrieved, loaded and executed at a time. In some example implementations, retrieval, loading and / or execution may be performed in parallel such that multiple instructions are retrieved, loaded, and / or executed together. Execution of the program code instructions may produce a computer-implemented process such that the instructions executed by the computer, processing circuitry or other programmable apparatus provide operations for implementing functions described herein.

[0206] Execution of instructions by processing unit, or storage of instructions in a computer-readable storage medium, supports combinations of operations for performing the specified functions. In this manner, a computer system (1) may include processing unit (20) and a computer-readable storage medium or memory (50) coupled to the processing circuitry, where the processing circuitry is configured to execute computer-readable program code (60) stored in the memory. It will also be understood that one or more functions, and combinations of functions, may be implemented by special purpose hardware-based computer systems and / or processing circuitry which perform the specified functions, or combinations of special purpose hardware and program code instructions.

[0207] Fig. 6 schematically shows an embodiment of the computer-implemented method for generating a synthetic medical representation in the form of a flow chart. The method comprises the steps:

[0208] (a) providing a pre-trained conditional generative model, wherein the pre-trained conditional generative model was trained to reconstruct a plurality of reference representations of an examination area of a plurality of examination objects, (b) providing a representation of the examination area of a new examination object,

[0209] (c) providing starting data,

[0210] (d) generating a synthetic representation of the examination area of the new examination object based on the starting data and one or more conditions using the pre -trained conditional generative model,

[0211] (e) generating a transformed synthetic representation based on the synthetic representation,

[0212] (f) quantifying a deviation between the representation of the examination area of the new examination object and the transformed synthetic representation,

[0213] (g) reducing the deviation by modifying the starting data and / or the one or more conditions, (h) repeating steps (d) to (g) until a stop criterion (S) is reached,

[0214] (i) outputting and / or storing the synthetic representation of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system.

Claims

CLAIMS1. A computer-implemented method comprising:(a) providing a pre-trained conditional generative model (CGM1). wherein the pre-trained conditional generative model (CGM1) was trained to reconstruct a plurality of reference representations (Rl, R2, R3, R4, R5, R6) of an examination area of a plurality of examination objects,(b) providing a representation (R) of the examination area of a new examination object,(c) providing starting data (SD),(d) generating a synthetic representation (SRT) of the examination area of the new examination object based on the starting data (SD) and one or more conditions using the pre-trained conditional generative model (CGM1).(e) generating a transformed synthetic representation (SRT) based on the synthetic representation (SR),(f) quantifying a deviation between the representation (R) of the examination area of the new examination object and the transformed synthetic representation (SRT),(g) reducing the deviation by modifying the starting data (SD) and / or the one or more conditions,(h) repeating steps (d) to (g) until a stop criterion is reached,(i) outputting and / or storing the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system.

2. The method of claim 1, wherein the pre-trained conditional generative model (CGM‘) was trained on training data (TD), wherein the training data (TD) comprised, for each examination object of the plurality of examination objects, a reference representation (Rl, R2, R3, R4, R5, R6) of the examination area of the examination object.

3. The method of claim 2, wherein the training data (TD) further comprised one or more annotations (Al, A4, A6) for at least a portion of the reference representation (Rl, R2, R3, R4, R5, R6) of the examination area of the plurality of examination objects, wherein the pre-trained conditional generative model (CGM‘) is or comprises a conditional diffusion model or a part thereof, wherein one or more conditions specified by the one or more annotations (Al, A4, A6) were used as conditions in training the conditional generative model (CGM), wherein the representation (R) of the examination area of the new examination object comprises an annotation (A), wherein one or more conditions derived from the annotation (A) are used as one or more conditions in generating the synthetic representation (SR) of the examination area of the examination object, wherein reducing the deviation by modifying the starting data (SD) and / or the one or more conditions comprises: reducing the deviation by modifying the staring data (SD) and / or one or more conditions derived from the annotation (A).

4. The method of any one of claims 1 to 3, wherein each examination object is a human, wherein the examination area is or comprises a liver, a kidney, a heart, a lung, a brain, a stomach, a bladder, a prostate, an intestine, an eye or a breast of the human or a part thereof.

5. The method of any one of claims 1 to 4, wherein each reference representation (Rl, R2, R3, R4, R5, R6) of the examination area of each examination object is a medical representation of the examination area of the examination object, wherein the representation (R) of the examination area of the new examination object is a medical representation of the examination area of the new examination object, and wherein the synthetic representation (SR) is a synthetic medical representation of the examination area of the new examination object.

6. The method of any one of claims 1 to 5, wherein each representation (R, Rl, R2, R3, R4, R5, R6) of the examination area of each examination object is a computer tomography image, an X-ray image, a magnetic resonance imaging image, a fluorescein angiography image, an optical coherence tomography image, a histological image, an ultrasound image, a fundus image, or a microscopic image, and the synthetic representation (SR) is a synthetic computer tomography image, a synthetic X-ray image, a synthetic magnetic resonance imaging image, a synthetic fluorescein angiography image, a synthetic optical coherence tomography image, a synthetic histological image, a synthetic ultrasound image, a synthetic fundus image, or a synthetic microscopic image.

7. The method of any one of claims 1 to 6, wherein each reference representation (Rl, R2, R3, R4, R5, R6) of the examination area of each examination object is a radiological representation of the examination area of the examination object, wherein the representation (R) of the examination area of the new examination object is a radiological representation of the examination area of the new examination object, and wherein the synthetic representation (SR) is a synthetic radiological representation of the examination area of the examination object.

8. The method of any one of claims 1 to 7, wherein the representation (R) of the examination area of the new examination object is a measured medical representation of the examination area of the new examination object after administration of a first amount of a contrast agent, and the synthetic representation (SR) is a synthetic medical representation of the examination area of the new examination object after administration of a second amount of the contrast agent, wherein the first amount and the second amount are different from each other.

9. The method of any one of claims 1 to 8, wherein the representation (R) of the examination area of the new examination object is a result of a radiological examination on the new examination object with a first radiation dose, and the synthetic representation (SR) is a synthetic medical representation of the examination area of the new examination object as a result of a radiological examination with a second radiation dose, wherein the first radiation dose and the second radiation dose are different from each other.

10. The method of any one of claims 1 to 9, wherein the representation (R) of the examination area of the new examination object comprises a measured representation of the examination area of the new examination object at a first time point before and / or after the application of a contrast agent, and the synthetic representation (SR) is a synthetic medical representation of the examination area of the new examination object at a second time point before or after the application of the contrast agent, wherein the first time point and the second time point are different from each other.

11. The method of any one of claims 1 to 9, wherein the synthetic representation (SR) is a synthetic representation of the examination area of the new examination object in real space.

12. The method of any one of claims 1 to 11, wherein the synthetic representation (SR) is a synthetic representation of the examination area of the new examination object in frequency space.

13. The method of any one of claims 1 to 12, wherein the synthetic representation (SR) is a synthetic representation of the examination area of the new examination object in projection space.

14. The method of any one of claims 1 to 13, wherein the stop criterion is: a predefined maximum number of steps has been performed, deviations between the transformed synthetic representation (SRT) and the representation (R) of the examination area of the new examination object can no longer be reduced by modifying the starting data (SD) and / or the one or more conditions, a predefined minimum of a loss function is reached, and / or an extreme value of another performance value is reached.

15. The method of any one of claims 1 to 14, comprising:(a) providing a pre-trained conditional generative model (CGM‘), wherein the pre-trained conditional generative model (CGM‘) was trained to reconstruct a plurality of reference representations (Rl, R2, R3, R4, R5, R6) of an examination area of a plurality of examination objects,(b) providing a representation (R) of the examination area of a new examination object, wherein the representation (R) comprises a radiologic representation of the examination area of the new examination object after application of a first amount of a contrast agent,(c) providing starting data (SD),(d) generating a synthetic representation (SR) of the examination area of the new examination object based on the starting data (SD) and one or more conditions using the pre-trained conditional generative model (CGM‘), wherein the synthetic representation (SR) is a synthetic radiologic representation of the examination area of the examination object after application of a second amount of the contrast agent, wherein the second amount is different from the first amount,(e) generating a transformed synthetic representation (SRT) based on the synthetic representation (SR), wherein the transformed synthetic representation (SRT) is a synthetic representation of the examination area of the examination object after application of the first amount of a contrast agent,(f) quantifying a deviation between the representation (R) of the examination area of the examination object and the transformed synthetic representation (SRT),(g) reducing the deviation by modifying the starting data (SD) and / or the one or more conditions,(h) repeating steps (d) to (g) until a stop criterion is reached,(i) outputting and / or storing the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation (SR) of theexamination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system.

16. The method of any one of claims 1 to 15, comprising:(a) providing a pre-trained conditional generative model (CGM1). wherein the pre-trained conditional generative model (CGM1) was trained to reconstruct a plurality of reference representations (Rl, R2, R3, R4, R5, R6) of an examination area of a plurality of examination objects,(b) providing a representation (R) of the examination area of a new examination object, wherein the representation (R) comprises a radiologic representation of the examination area of the new examination object using a first radiation dose,(c) providing starting data (SD),(d) generating a synthetic representation (SR) of the examination area of the new examination object based on the starting data (SD) and one or more conditions using the pre-trained conditional generative model (CGM1). wherein the synthetic representation (SR) is a synthetic radiologic representation of the examination area of the new examination object using a second radiation dose, wherein the second radiation dose is different from the first radiation dose,(e) generating a transformed synthetic representation (SRT) based on the synthetic representation (SR), wherein the transformed synthetic representation (SRT) is a synthetic representation of the examination area of the new examination object using the first radiation dose,(f) quantifying a deviation between the representation (R) of the examination area of the new examination object and the transformed synthetic representation (SRT),(g) reducing the deviation by modifying the starting data (SD) and / or the one or more conditions,(h) repeating steps (d) to (g) until a stop criterion is reached,(i) outputting and / or storing the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system.

17. The method of any one of claims 1 to 16, comprising:(a) providing a pre-trained conditional generative model (CGM‘), wherein the pre-trained conditional generative model (CGM1) was trained to reconstruct a plurality of reference representations (Rl, R2, R3, R4, R5, R6) of an examination area of a plurality of examination objects,(b) providing a representation (R) of the examination area of a new examination object, wherein the representation (R) comprises a radiologic representation of the examination area of the new examination object at a first time point before or after the application of a contrast agent,(c) providing starting data (SD),(d) generating a synthetic representation (SR) of the examination area of the new examination object based on the starting data (SD) and one or more conditions using the pre-trained conditional generative model (CGM1). wherein the synthetic representation (SR) is a synthetic radiologic representation of the examination area of the new examination object at a second timepoint before or after the application of a contrast agent, wherein the second time point and the first time point are different time points,(e) generating a transformed synthetic representation (SRT) based on the synthetic representation (SR), wherein the transformed synthetic representation (SRT) is a synthetic representation of the examination area of the new examination object at the first time point before or after the application of a contrast agent,(f) quantifying a deviation between the representation (R) of the examination area of the new examination object and the transformed synthetic representation (SRT),(g) reducing the deviation by modifying the starting data (SD) and / or the one or more conditions,(h) repeating steps (d) to (g) until a stop criterion is reached,(i) outputting and / or storing the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system.

18. A computer system (1) comprising: a processing unit (20); and a memory (50) storing an application program (60) configured to perform, when executed by the processing unit (20), an operation, the operation comprising:(a) providing a pre-trained conditional generative model (CGM‘), wherein the pre-trained conditional generative model (CGM‘) was trained to reconstruct a plurality of reference representations (Rl, R2, R3, R4, R5, R6) of an examination area of a plurality of examination objects,(b) providing a representation (R) of the examination area of a new examination object,(c) providing starting data (SD),(d) generating a synthetic representation (SR) of the examination area of the new examination object based on the starting data (SD) and one or more conditions using the pre-trained conditional generative model (CGM‘),(e) generating a transformed synthetic representation (SRT) based on the synthetic representation (SR),(f) quantifying a deviation between the representation (R) of the examination area of the new examination object and the transformed synthetic representation (SRT),(g) reducing the deviation by modifying the starting data (SD) and / or the one or more conditions,(h) repeating steps (d) to (g) until a stop criterion is reached,(i) outputting and / or storing the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system.

19. A non-transitory computer readable storage medium having stored thereon software instructions that, when executed by a processing unit (20) of a computer system (1), cause the computer system (1) to perform the following steps:(a) providing a pre-trained conditional generative model (CGM1). wherein the pre-trained conditional generative model (CGM1) was trained to reconstruct a plurality of reference representations (Rl, R2, R3, R4, R5, R6) of an examination area of a plurality of examination objects,(b) providing a representation (R) of the examination area of a new examination object,(c) providing starting data (SD),(d) generating a synthetic representation (SR) of the examination area of the new examination object based on the starting data (SD) and one or more conditions using the pre-trained conditional generative model (CGM1).(e) generating a transformed synthetic representation (SRT) based on the synthetic representation (SR),(f) quantifying a deviation between the representation (R) of the examination area of the new examination object and the transformed synthetic representation (SRT),(g) reducing the deviation by modifying the starting data (SD) and / or the one or more conditions,(h) repeating steps (d) to (g) until a stop criterion is reached,(i) outputting and / or storing the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom, and / or transmitting the synthetic representation (SR) of the examination area of the new examination object or a synthetic medical image of the examination area of the new examination object generated therefrom to a separate computer system.

Images