Generation of synthetic medical images
By training machine learning models to generate synthetic medical images, the problems of discomfort and low-quality images caused by prolonged patient immobility during dynamic imaging examinations have been solved, achieving efficient and high-quality image generation and improving diagnostic efficiency.
Patent Information
- Application Number
- CN202580027199.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-03-19
- Filing Date
- 2025-03-10
- Publication Date
- 2026-11-10
AI Technical Summary
During dynamic medical imaging examinations, patients need to remain still for extended periods to reduce image motion artifacts, which can cause discomfort. Additionally, low-quality images may appear that cannot be replaced without repeating the examination, thus affecting diagnostic efficiency.
By employing an image encoder and a conditional generation model, synthetic medical images are generated through training a machine learning model. By utilizing medical image datasets from multiple time points, bias is reduced, resulting in high-quality synthetic images.
Accelerate the dynamic examination process, reduce patient discomfort, improve image quality, avoid repeated examinations, and improve diagnostic efficiency.
Smart Images

Figure CN123076899A_ABST
Abstract
Description
Technical Field
[0001] The systems, methods, and computer programs disclosed herein relate to training a machine learning model and using the trained machine learning model to generate synthetic medical images of an examination region of an object, the synthetic medical images representing the examination region at a point in time during a dynamic examination of the examination region, the generation being based on one or more medical images representing the examination region at another point in time or at multiple other points in time. Background Technology
[0002] Dynamic medical imaging refers to generating medical images of the examination area of an object at different points in time, so as to visualize the changes of the examination area over time.
[0003] Dynamic medical imaging can be used for diagnostic purposes. One example is the use of dynamic contrast-enhanced magnetic resonance imaging (MRI) with hepatobiliary contrast agents to detect and differentiate focal liver lesions.
[0004] Hepatobiliary contrast agents such as Primovist® can be used to detect tumors in the liver. Healthy liver tissue is primarily supplied by the portal vein, while most primary tumors are supplied by the hepatic artery. After intravenous bolus injection of the contrast agent, a time delay can be observed between the signal enhancement of healthy liver parenchyma and that of tumors.
[0005] Besides malignant tumors, benign lesions such as cysts, hemangiomas, and focal nodular hyperplasia are frequently found in the liver. To develop an appropriate treatment plan, it is essential to differentiate these benign lesions from malignant tumors. Primovist® can be used to detect benign and malignant focal liver lesions. It utilizes T1-weighted MRI to provide information about the characteristics of these lesions. The different blood supply to the liver and tumors, as well as the temporal progression of contrast enhancement, are used for differentiation.
[0006] Therefore, tracking the distribution of contrast agents over time provides a good opportunity for the detection and differential diagnosis of focal liver lesions.
[0007] The use of dynamic contrast-enhanced magnetic resonance imaging (MRI) with hepatobiliary contrast agents for the detection and differential diagnosis of focal liver lesions is described herein only as one example of a dynamic examination; many other examples exist. The examples given should not be construed as limiting the scope of this disclosure to that example.
[0008] Dynamic examinations can take a relatively long time. During this time, the patient should largely avoid movement to minimize motion artifacts in the medical images. Prolonged restriction of movement can be uncomfortable for the patient. It would be desirable to be able to expedite dynamic examinations.
[0009] Furthermore, during dynamic examination of the examination area of the subject, one or more medical images of the examination area may become of lower quality due to movement of the subject and / or other reasons. Therefore, it would be desirable to be able to replace poor-quality medical images without having to repeat the entire examination. Summary of the Invention
[0010] These and other tasks are addressed by the subject matter of the independent claims of this disclosure. Preferred embodiments are defined in the dependent claims, the specification, and the drawings.
[0011] In a first aspect, this disclosure relates to a computer-implemented method, comprising: - Provides multiple datasets for multiple examination subjects, each dataset including at least two medical images: a first medical image and a second medical image, wherein the first medical image represents the examination area of the examination subject at a first time point, and the second medical image represents the examination area of the examination subject at a second time point, wherein the first time point and the second time point are different time points. - Provides an image encoder, wherein the image encoder is configured to generate image embeddings based on medical images. - For each dataset: use the image encoder to generate the image embedding of the first medical image. - Provides a conditional generation model, wherein the conditional generation model is configured to generate a reconstructed medical image based on the medical image, conditions, and model parameters. - Train the conditional generation model, wherein for each dataset, the training includes: A reconstructed second medical image is generated based on the image embedding of the first medical image and the second medical image, wherein the image embedding of the first medical image is used as a condition in the generation of the reconstructed second medical image. Determine the deviation between the second medical image and the reconstructed second medical image. The bias can be reduced by modifying the model parameters of the conditional generation model. - Receive at least one medical image of a new subject for examination, wherein the at least one medical image represents the examination area of the new subject at a first time point. - Using the image encoder, generate at least one image embedding based on at least one medical image of the new examination subject. - Generate synthetic medical images using the at least one image embedding as conditional input to the trained conditional generation model. - Output and / or store the synthetic medical images, and / or transmit the synthetic medical images to a separate computer system.
[0012] On the other hand, this disclosure provides a computer system comprising: Processing unit; and A memory storing a computer program configured to perform an operation when implemented by the processing unit, the operation including: - Provides multiple datasets for multiple examination subjects, each dataset including at least two medical images: a first medical image and a second medical image, wherein the first medical image represents the examination area of the examination subject at a first time point, and the second medical image represents the examination area of the examination subject at a second time point, wherein the first time point and the second time point are different time points. - Provides an image encoder, wherein the image encoder is configured to generate image embeddings based on medical images. - For each dataset: use the image encoder to generate the image embedding of the first medical image. - Provides a conditional generation model, wherein the conditional generation model is configured to generate a reconstructed medical image based on the medical image, conditions, and model parameters. - Train the conditional generation model, wherein for each dataset, the training includes: A reconstructed second medical image is generated based on the image embedding of the first medical image and the second medical image, wherein the image embedding of the first medical image is used as a condition in the generation of the reconstructed second medical image. Determine the deviation between the second medical image and the reconstructed second medical image. The bias can be reduced by modifying the model parameters of the conditional generation model. - Receive at least one medical image of a new subject for examination, wherein the at least one medical image represents the examination area of the new subject at a first time point. - Using the image encoder, generate at least one image embedding based on at least one medical image of the new examination subject. - Generate synthetic medical images using the at least one image embedding as conditional input to the trained conditional generation model. - Output and / or store the synthetic medical images, and / or transmit the synthetic medical images to a separate computer system.
[0013] On the other hand, this disclosure provides a non-transitory computer-readable storage medium on which a computer program is stored, which, when executed by a processing unit of a computer system, causes the computer system to perform the following steps: - Provides multiple datasets for multiple examination subjects, each dataset including at least two medical images: a first medical image and a second medical image, wherein the first medical image represents the examination area of the examination subject at a first time point, and the second medical image represents the examination area of the examination subject at a second time point, wherein the first time point and the second time point are different time points. - Provides an image encoder, wherein the image encoder is configured to generate image embeddings based on medical images. - For each dataset: use the image encoder to generate the image embedding of the first medical image. - Provides a conditional generation model, wherein the conditional generation model is configured to generate a reconstructed medical image based on the medical image, conditions, and model parameters. - Train the conditional generation model, wherein for each dataset, the training includes: A reconstructed second medical image is generated based on the image embedding of the first medical image and the second medical image, wherein the image embedding of the first medical image is used as a condition in the generation of the reconstructed second medical image. Determine the deviation between the second medical image and the reconstructed second medical image. The bias can be reduced by modifying the model parameters of the conditional generation model. - Receive at least one medical image of a new subject for examination, wherein the at least one medical image represents the examination area of the new subject at a first time point. - Using the image encoder, generate at least one image embedding based on at least one medical image of the new examination subject. - Generate synthetic medical images using the at least one image embedding as conditional input to the trained conditional generation model. - Output and / or store the synthetic medical images, and / or transmit the synthetic medical images to a separate computer system.
[0014] On the other hand, this disclosure relates to the use of contrast agents, and / or the use of contrast agents in dynamic examination of the examination area of the subject, said dynamic examination including: - Receive one or more medical images, wherein the one or more medical images represent the examination area of the subject at one or more time points before and / or after the application of the contrast agent. - Provide an image encoder, wherein the image encoder is configured to generate an image embedding based on the one or more medical images. - Using the image encoder, generate at least one image embedding based on the one or more medical images. - Provides a trained conditional generation model, wherein the trained conditional generation model is configured and trained to generate synthetic medical images based on image embeddings as conditions. - Generate synthetic medical images using the at least one image embedding as conditional input to the trained conditional generation model. - Output and / or store the synthetic medical images, and / or transmit the synthetic medical images to a separate computer system. The training of the trained conditional generation model includes: - Provides multiple datasets of multiple reference examination subjects, each dataset including at least two reference medical images: one or more first reference medical images and a second reference medical image, wherein the one or more first reference medical images represent the examination area of the reference examination subject at one or more time points before and / or after the application of the contrast agent, and the second reference medical image represents the examination area of the reference examination subject at another time point before and / or after the application of the contrast agent. - For each dataset: use the image encoder to generate one or more reference image embeddings of the one or more first reference medical images. - Train the conditional generation model, wherein for each dataset, the training includes: A reconstructed second reference medical image is generated based on the one or more reference image embeddings of the one or more first reference medical images and the second reference medical image, wherein the one or more reference image embeddings of the one or more first reference medical images are used as conditions in the generation of the reconstructed second reference medical image. Determine the deviation between the second reference medical image and the reconstructed second reference medical image. The bias is reduced by modifying the model parameters of the conditional generation model.
[0015] On the other hand, this disclosure provides a kit including a contrast agent and a computer program, which, when implemented by a processing unit of a computer system, causes the computer system to perform the following steps: - Receive one or more medical images, wherein the one or more medical images represent the examination area of the subject at one or more time points before and / or after the application of the contrast agent. - Provide an image encoder, wherein the image encoder is configured to generate an image embedding based on the one or more medical images. - Using the image encoder, generate at least one image embedding based on the one or more medical images. - Provides a trained conditional generation model, wherein the trained conditional generation model is trained to generate synthetic medical images based on image embeddings as conditions. - Generate synthetic medical images using the at least one image embedding as conditional input to the trained conditional generation model. - Output and / or store the synthetic medical images, and / or transmit the synthetic medical images to a separate computer system. The training of the trained conditional generation model includes: - Provides multiple datasets of multiple reference examination subjects, each dataset including at least two reference medical images: one or more first reference medical images and a second reference medical image, wherein the one or more first reference medical images represent the examination area of the reference examination subject at one or more time points before and / or after the application of the contrast agent, and the second reference medical image represents the examination area of the reference examination subject at another time point before and / or after the application of the contrast agent. - For each dataset: use the image encoder to generate one or more reference image embeddings of the one or more first reference medical images. - Train the conditional generation model, wherein for each dataset, the training includes: A reconstructed second reference medical image is generated based on the one or more reference image embeddings of the one or more first reference medical images and the second reference medical image, wherein the one or more reference image embeddings of the one or more first reference medical images are used as conditions in the generation of the reconstructed second reference medical image. Determine the deviation between the second reference medical image and the reconstructed second reference medical image. The bias is reduced by modifying the model parameters of the conditional generation model.
[0016] Other aspects are disclosed in the specification. Attached Figure Description
[0017] Figure 1 An implementation scheme for training a conditional generative model is illustrated schematically.
[0018] Figure 2 Another implementation scheme for training a conditional generative model is illustrated schematically.
[0019] Figure 3 A schematic example of how to generate image embeddings from medical images is shown.
[0020] Figure 4 A schematic embodiment of how multiple parts of an image can be masked is shown.
[0021] Figure 5 A schematic embodiment is shown that generates several image embeddings and combines these image embeddings into a single image embedding.
[0022] Figure 6 An example of masking blocks and / or image tokens is shown.
[0023] Figure 7 Another embodiment of masking blocks and / or image terms is shown.
[0024] Figure 8 Another embodiment of generating image embeddings based on multiple medical images is shown.
[0025] Figure 9 An implementation scheme for generating synthetic medical images based on medical images is shown.
[0026] Figure 10 An implementation scheme for generating synthetic medical images based on several medical images is shown.
[0027] Figure 11 Another implementation scheme for generating synthetic medical images based on several medical images is shown.
[0028] Figure 12 An implementation scheme of the computer-based method of this disclosure is illustrated in the form of a flowchart.
[0029] Figure 13 Computer systems according to some exemplary embodiments of the present disclosure are illustrated in more detail. Detailed Implementation
[0030] Several aspects of this disclosure will be set forth in more detail below, without distinguishing between the various aspects of this disclosure (computer-implemented methods, computer systems, computer-readable storage media, uses, contrast agents used, and assemblies). Rather, the following description is intended to apply similarly to all aspects of this disclosure, regardless of the context in which they appear (computer-implemented methods, computer systems, computer-readable storage media, uses, contrast agents used, and assemblies).
[0031] The fact that the steps are stated in one order in this specification or claims does not necessarily mean that this disclosure is limited to the stated order. Rather, it is conceivable that these steps may also be performed in a different order or in parallel with each other, unless, for example, one step builds upon another, which requires the built-up step to be performed subsequently (however, this is clear in individual cases). Therefore, the stated order is an exemplary embodiment of this disclosure.
[0032] As used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” As used in the specification and claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly indicates otherwise. The term “an” or similar language is used only when intended to refer to only one item. Furthermore, as used herein, the terms “have,” “possess,” “have,” etc., are intended to be open-ended terms. Additionally, the phrase “based on” is intended to mean “at least partially based on” unless otherwise clearly stated.
[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.
[0034] Some embodiments of this disclosure will be described more fully below with reference to the accompanying drawings, which illustrate some, but not all, embodiments of this disclosure. In fact, various embodiments of this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of this disclosure to those skilled in the art.
[0035] This disclosure provides an apparatus for generating synthetic medical images of the examination area of an examination subject.
[0036] The term "synthetic" means that a synthetic medical image is not the (direct) result of physical measurements of the real object being examined, but rather a synthetic medical image generated by a machine learning model. The synonym for "synthetic" is "artificial." However, synthetic medical images can be based on one or more measured medical images; that is, a machine learning model can be configured and trained to generate synthetic medical images based on one or more measured medical images (and / or other / further data).
[0037] The "object of inspection" is typically a living organism, such as a mammal, like a human. In one embodiment of this disclosure, the object of inspection is a human.
[0038] The "examination area" is a part of the object being examined, such as an organ or a part of an organ or multiple organs, or another part of the object being examined.
[0039] For example, the area to be examined may be the liver, kidneys, heart, lungs, brain, stomach, bladder, prostate, intestines, thyroid, eyes, pancreas, breast, uterus, or part of said sites or another part of the body of a mammal (e.g., human).
[0040] In one embodiment of this disclosure, the examination area includes the liver or a portion of the liver, or the examination area is the liver or a portion of the liver of a mammal (e.g., a human).
[0041] In another embodiment, the area to be examined includes the brain or a portion of the brain, or the area to be examined is the brain or a portion of the brain of a mammal (e.g., a human).
[0042] In another embodiment, the examination area includes the heart or a portion of the heart, or the examination area is the heart or a portion of the heart of a mammal (e.g., a human).
[0043] In another embodiment, the examination area includes the chest or a portion of the chest, or the examination area is the chest or a portion of the chest of a mammal (e.g., a human).
[0044] In another embodiment, the examination area includes the stomach or a portion of the stomach, or the examination area is the stomach or a portion of the stomach of a mammal (e.g., a human).
[0045] In another embodiment, the examination area includes the pancreas or a portion thereof, or the examination area is the pancreas or a portion thereof of a mammal (e.g., a human).
[0046] In another embodiment, the examination area includes the kidney or a portion of the kidney, or the examination area is the kidney or a portion of the kidney of a mammal (e.g., a human).
[0047] In another embodiment, the examination area includes one or both lungs or a portion of a lung of a mammal (e.g., a human).
[0048] In another embodiment, the area to be examined includes the thyroid gland or a portion thereof in a mammal (e.g., a human).
[0049] In another embodiment, the area to be examined includes the eye or part of the eye of a mammal (e.g., a human).
[0050] In another embodiment, the examination area includes a mammary gland or a portion thereof, or the examination area is a mammary gland or a portion thereof of a female mammal (e.g., a human woman).
[0051] In another embodiment, the examination area includes the uterus or a portion thereof, or the examination area is the uterus or a portion thereof of a female mammal (e.g., a human woman).
[0052] In another embodiment, the examination area includes the prostate or a portion thereof, or the examination area is the prostate or a portion thereof of a male mammal (e.g., a male human).
[0053] In one embodiment of this disclosure, the term "image" as used herein refers to a data structure representing the spatial distribution of a physical signal. The spatial distribution can be of any dimension, such as 2D, 3D, 4D, or any higher dimension. The spatial distribution can be of any shape, such as forming a grid and thereby defining pixels or voxels; the grid can be irregular or regular. The physical signal can be any signal, such as proton density, tissue echogenicity, tissue radiolucency, blood flow-related measurements, information about rotating hydrogen nuclei in a magnetic field, color, grayscale level, depth, surface or volume occupancy, such that the image can be a 2D or 3D RGB / grayscale / depth image, or a 3D surface / volume occupancy model. Images are typically composed of discrete image elements (e.g., pixels in a 2D image, voxels in a 3D image, and doxels in a 4D image).
[0054] Several aspects of this disclosure are described herein primarily using embodiments of images representing the inspection area of an object in real space. However, it should be noted that this disclosure can also be applied to the representation of the inspection area in other spaces, such as frequency space or projection space. In this regard, the term "image" should be interpreted broadly.
[0055] In real-space representation (also referred to as real-space representation in this specification), the area under examination is typically represented by a large number of image elements (e.g., pixels, voxels, or 4D pixels), which may be arranged, for example, in a raster pattern, where each image element represents a portion of the area under examination, and each image element may be assigned a color value or grayscale value. The color value or grayscale value represents the signal intensity, such as the attenuation of X-rays. The DICOM format is widely used in radiology for storing and processing real-space representations. DICOM (Digital Imaging and Communications in Medicine) is an open standard for storing and exchanging information in medical image data management.
[0056] In the frequency-space representation (also referred to as frequency-space representation in this specification), the region of examination is represented by a superposition of fundamental oscillations. For example, the region of examination can be represented by a sum of sine and cosine functions with different amplitudes, frequencies, and phases. These amplitudes and phases can be plotted as functions of frequency, for example, in two-dimensional or three-dimensional plots. Typically, the lowest frequency (origin) is located at the center. The further away from this center, the higher the frequency. Each frequency can be assigned an amplitude (representing the frequency in the frequency-space representation) and a phase (indicating the degree of deviation of the corresponding oscillation towards a sinusoidal or cosine oscillation).
[0057] A representation in real space can be transformed into a representation in frequency space, for example, through a Fourier transform. Conversely, a representation in frequency space can be transformed into a representation in real space, for example, through an inverse Fourier transform.
[0058] Detailed information about real space representations and frequency space representations and their respective conversions is described in many publications, see, for example, https: / / see.stanford.edu / materials / lsoftaee261 / book-fall-07.pdf.
[0059] The representation of the examined area in projection space is typically the result of a computed tomography (CT) scan prior to image reconstruction. In other words, the raw data obtained in a CT scan can be understood as a representation in projection space. In CT, the intensity or attenuation of X-rays is measured as they pass through the object being examined. Based on this, projection values can be calculated. In the second step, computer-aided reconstruction transforms the object information encoded by the projection into a true spatial representation. This reconstruction can be achieved using the Radon transform. The Radon transform describes the relationship between an unknown object being examined and its associated projection.
[0060] Detailed information on the transformation of projected data to a real-space representation is described in numerous publications; see, for example, K. Catch. The Radon Transformation and Its Application in Tomography,Journal of Physics Conference Series 1903(1):012066.
[0061] A "medical image" is a representation of the human body or a part thereof, or of the animal body or a part thereof. Medical images can be used for purposes such as diagnosis and / or treatment.
[0062] Techniques used to generate medical images include X-ray photography, computed tomography, fluoroscopy, magnetic resonance imaging, ultrasound, endoscopy, elastography, tactile imaging, thermal imaging, microscopy, positron emission tomography, optical coherence tomography, fundus photography, and others.
[0063] Examples of medical images include CT (computed tomography) scans, X-ray images, MRI (magnetic resonance imaging) scans, PET (positron emission tomography) scans, fluorescein angiography images, OCT (optical coherence tomography) scans, histological images, ultrasound images, fundus images, and / or others.
[0064] In one embodiment of this disclosure, a synthetic medical image is a synthetic radiographic image. "Radiology" is a branch of medicine that involves the application of electromagnetic radiation and mechanical waves (including, for example, diagnostic ultrasound) for diagnostic, therapeutic, and / or scientific purposes. Other ionizing radiation, such as gamma rays or electrons, is used in addition to X-rays. Since the primary purpose is imaging, other imaging procedures (such as ultrasound and magnetic resonance imaging (MRI)) are also included in radiology, although ionizing radiation is not used in these procedures. Therefore, the term "radiology" as used in this disclosure specifically includes the following examination procedures: computed tomography, magnetic resonance imaging, ultrasound, and positron emission tomography.
[0065] Synthetic radiographic images can be a representation of the examined area of an object with or without the application of contrast agents.
[0066] "Contrast agent" is a substance or mixture of substances that improves the depiction of body structure and function during radiological examinations.
[0067] In computed tomography (CT) scans, iodine-containing solutions are commonly used as contrast agents. In magnetic resonance imaging (MRI), superparamagnetic materials (e.g., iron oxide nanoparticles, superparamagnetic iron-platinum particles (SIPP)) or paramagnetic materials (e.g., gadolinium chelates, manganese chelates, hafnium chelates) are commonly used as contrast agents. In ultrasound examinations, a liquid containing aerated microbubbles is typically administered intravenously. In positron emission tomography (PET), a radioactive tracer is used as a contrast agent. The contrast in PET images is caused by the differential uptake of the radioactive tracer in different tissues or organs. A radioactive tracer is a radioactive substance injected into the subject being examined. The radioactive tracer emits positrons. When a positron collides with an electron in the area of the subject being examined, the two particles annihilate, producing two gamma rays emitted in opposite directions. These gamma rays are then detected by a PET scanner, allowing the creation of detailed images of the body's internal functions.
[0068] Examples of contrast agents can be found in the literature (see, for example, ASL Jascinth et al.): Contrast Agents in computed tomography: A Review Journal of Applied Dental and Medical Sciences, 2016, Vol. 2, No. 2, pp. 143-149; H. Lusic et al.: X-ray-Computed Tomography Contrast Agents , Chem . Rev . 2013, 113, 3, 1641-1666; https: / / www.radiology.wisc.edu / wp-content / uploads / 2017 / 10 / contrast-agents-tutorial.pdf, MR Nouh et al.: Radiographic and magnetic resonances contrast agents: Essentials and tips for safe practices , World J Radiol. 28 Sep 2017; 9(9): 339-349; LC Abonyi et al: Intravascular Contrast Media in Radiography: Historical Development & Review of Risk Factors for Adverse Reactions , South American Journal of Clinical Research, 2016, Vol. 3, No. 1, 1-10; ACR Manual on Contrast Media, 2020, ISBN: 978-1-55903-012-0; A. Ignee et al.: Ultrasound contrast agents, Endosc Ultrasound. November-December 2016; 5(6): 355-362; J. Trotter People: Positron Emission Tomography (PET) / Computed Tomography (CT) Imaging in Radiation Therapy Treatment Planning: A Review of PET Imaging Tracers and Methods to Incorporate PET / CT , Advances in Radiation Oncology(2023) 8, 101212).
[0069] In one embodiment of this disclosure, the synthetic medical image is a synthetic MRI image. The synthetic MRI image may be generated based on one or more measured MRI images. The one or more measured MRI images may represent the liver or a portion of the liver at one or more time points before and / or after the application of an MRI contrast agent. The synthetic MRI image may represent the liver or a portion of the liver at another time point before or after the application of an MRI contrast agent. The MRI contrast agent may be or includes a hepatobiliary contrast agent. A "hepatobiliary contrast agent" is characterized by being specifically taken up by hepatocytes (liver cells), accumulating in functional tissue (parenchyma), and enhancing the contrast of healthy liver tissue. One example of a hepatobiliary MRI contrast agent is disodium gadoxetate (Gd-EOB-DTPA disodium), described in U.S. Patent No. 6,039,931A and traded under the name Primovist. ® and Eovist ® Commercially available. Other hepatobiliary contrast agents are described in particular in WO2022 / 194777.
[0070] The one or more measured MRI images may represent the liver or a portion of the liver during one or more of the following periods before and / or after the administration of hepatobiliary MRI contrast agents: plain phase, arterial phase, portal venous phase, transitional phase, and / or hepatobiliary phase. Synthetic MRI images may represent the liver during periods not represented by any of the measured MRI images. For example, the one or more measured MRI images may represent the liver or a portion of the liver during the plain phase, arterial phase, portal venous phase, and / or transitional phase before and / or after the administration of hepatobiliary MRI contrast agents, and synthetic MRI images may represent the liver or a portion of the liver during the hepatobiliary phase after the administration of hepatobiliary MRI contrast agents. For example, the one or more measured MRI images may represent the liver or a portion of the liver during the arterial phase, portal venous phase, transitional phase, and / or hepatobiliary phase after the administration of hepatobiliary MRI contrast agents, and synthetic MRI images may represent the liver or a portion of the liver during the plain phase before the administration of hepatobiliary MRI contrast agents.
[0071] The period mentioned above, for example, is described by CL Hui et al.: Patterns of enhancement in the hepatobiliary phase of gadoxetic acid-enhanced MRI, Br J Radiol., 2020, 93:20190989.
[0072] In another embodiment of this disclosure, the synthetic medical image is a synthetic CT image.
[0073] In another embodiment of this disclosure, the synthetic medical image is a synthetic PET image.
[0074] Synthetic medical images are generated using trained machine learning models.
[0075] The "machine learning model" used in this article can be understood as a data processing architecture implemented by a computer. A machine learning model receives input data and provides output data based on that input data and the model's parameters (model parameters). The machine learning model can learn the relationship between the input and output data through training. During training, the parameters of the machine learning model can be adjusted to provide the desired output for a given input.
[0076] The process of training a machine learning model involves feeding training data to a machine learning algorithm (i.e., a learning algorithm) for learning. The term "trained machine learning model" refers to the model output produced by the training process. The training data typically contains correct answers, which is called the target. The learning algorithm finds patterns in the training data that map the input data to the target, and it outputs a trained machine learning model that captures these patterns.
[0077] During training, input data is fed into the machine learning model, which then generates output. This output is compared to a (known) target. The parameters of the machine learning model are then modified to reduce the deviation between the output and the (known) target to a (defined) minimum.
[0078] Typically, a loss function can be used for training, where the loss function quantifies the deviation between the output and the target.
[0079] The goal of the training process can be to modify (adjust) the parameters of a machine learning model to reduce the loss to a (defined) minimum. This can be done during the optimization process, such as gradient descent.
[0080] The machine learning models disclosed herein are, or include, conditional generative models.
[0081] A generative model is a type of machine learning model designed to learn and generate new data similar to the training data on which it was trained. Generative models capture the underlying distribution of the training data and can generate samples from that distribution.
[0082] A "conditional generative model" is a type of generative model that generates data (in this case, synthetic medical images) given certain conditions or constraints. Conditional generative models receive additional input in the form of conditions that guide the image generation process. Typically, these conditions can be anything that provides some context for the generation process, such as category labels, text descriptions, another image, or any other information. In the case of this disclosure, one or more image embeddings are used as conditions.
[0083] In one embodiment of this disclosure, the conditional generation model is or includes a diffusion model.
[0084] Diffusion models focus on modeling the gradual evolution of data distributions from a "simple" starting point to a "more complex" distribution. The underlying concept of a diffusion model is to transform a simple and easily sampled distribution (typically a Gaussian distribution) into a more complex data distribution of interest. This transformation is achieved through a series of reversible operations. Once the model has learned the transformation process, it can generate new samples by starting from a point in a simple distribution and gradually "diffusion" it to the desired complex data distribution.
[0085] Diffusion models typically include noise-adding models and noise-reducing models.
[0086] Noisy models typically consist of multiple noisy stages. A noisy model is configured to receive input data (e.g., an image) and generate noisy data in response to the received input data. The noisy model introduces noise into the input data over several stages or "time steps". T The input data is then blurred. The noise-adding model may or may include a finite number of steps. T Or an infinite number of steps ( The noisy model can have the same weights / architecture for all time steps, or different weights / architecture for each time step. The number of time steps can be global (i.e., the same for all pixels in the image) or local (e.g., each pixel in the image can have a different time step). The noisy model can be based on different types of noise, such as noise sampled from a Gaussian distribution, and noise originating from blurring or masking.
[0087] The denoising model is configured to reconstruct the input data from the noisy data. The denoising model is configured to produce samples that match the input data after multiple stages.
[0088] For example, a diffusion model can include Markov chains at the noisy and / or denoising models. The diffusion model can be implemented in discrete time, for example, where each layer corresponds to a time step. The diffusion model can also be implemented at arbitrary depths (e.g., continuous) of time.
[0089] Diffusion models are conceptually similar to variational autoencoders (VAEs), whose structure and loss function provide efficient training for models of arbitrary depth (e.g., infinite depth). Diffusion models can be trained, for example, using variational inference.
[0090] The diffusion model can be a latent diffusion model (LDM). In such a model, in the case of an image, the diffusion method is not performed in the real space (e.g., pixel space, voxel space, or four-dimensional pixel space, depending on the case), but in a so-called latent space based on the image representation (usually a compressed representation) (see, for example, R. Rombach et al.). High-Resolution Image Synthesis with Latent Diffusion Models , arXiv:2112.10752v2).
[0091] The diffusion model can be a denoised diffusion probability model (DDPM). DDPM is a type of generative model that iteratively adds noise to input data (e.g., an image or compressed representation) and then learns to denoise the noisy signal to generate new samples (see, for example, J. Ho et al.). Denoising Diffusion Probabilistic Models ,arXiv:2006.11239v2) to work.
[0092] Diffusion models can be score-based generative models (SGM). In SGM, the data is perturbed by random Gaussian noise of varying magnitudes. Samples are generated in the direction of reducing noise levels using the gradient of the log probability density as the score function, and the model is trained by estimating the score function of the distribution of the noisy data (see, for example, Y. Song et al.). Score-Based Generative Modeling through Stochastic Differential Equations , arXiv:2011.13456v2).
[0093] The diffusion model can be a denoised diffusion implicit model (DDIM) (see, for example, J. Song et al.). Denoising Diffusion Implicit Models(arXiv:2010.02502v4). A key drawback of DDPM is that they require multiple iterations to produce high-quality samples. For DDPM, this is because the generation process (from noise to data) approximates the inverse of the forward diffusion process (from data to noise), which can have thousands of steps: all steps need to be iterated to produce a single sample. DDIM is an implicit probabilistic model closely related to DDPM because they are trained using the same objective function. DDIM allows for faster sampling while maintaining an equivalent training objective. They achieve this by estimating the cumulative result of multiple Markov chain steps and summing them all at once. DDIM constructs a class of non-Markov diffusion processes, which makes sampling from the reverse process faster. This modification in the forward process preserves the objective of DDPM and allows for deterministic encoding of images as noise maps.
[0094] Unlike DDPM, DDIM is able to control image composition due to the flexibility of the latent space (attribute operations) (see, for example, K. Preechakul et al.). Diffusion autoencoders: Toward a meaningful and decodable representation (arXiv:2111.15640v3). Using DDIM, the generation process can be deterministically reversed to obtain the noise map x. T DDIM represents the latent variables or encodings of a given image x0. In this context, DDIM can be considered an image decoder that decodes the latent code x. T Decode back the input image. This process can produce a very accurate reconstruction; however, x T It still does not include high-level semantics (which should be expected from meaningful representations).
[0095] In one embodiment of this disclosure, the conditional generation model is or includes a conditional diffusion model.
[0096] In conditional diffusion models, conditions are used to denoise the latent data and reconstruct the input data (see, for example, P. Dhariwal, A. Nichol: " Diffusion models beat GANs on image synthesis (arXiv:2105.05233v4). One advantage of using information-rich representations to conditionally control the diffusion model is a more efficient denoising process.
[0097] Typically, such conditions can be based on text (e.g., text-to-image), images, audio data, or other information. In the context of this disclosure, image embeddings of medical images are used as conditions for generating synthetic medical images.
[0098] The conditional generative model is first trained using training data. The training data is generated based on multiple datasets from multiple inspected objects.
[0099] The term "multiple" means more than ten, such as more than one hundred, or even more than one thousand.
[0100] Each dataset includes at least two medical images: a first medical image and a second medical image. The first medical image represents the examination area of the subject at a first time point. The second medical image represents the examination area of the subject at a second time point. The first time point and the second time point are different time points.
[0101] For each dataset, the area examined is typically the same. For each dataset, the object examined can be different; however, it is also possible that medical images from one or more datasets represent the same area examined for the same object.
[0102] The second time point can be after the first time point; however, the second time point can also be before the first time point.
[0103] In one embodiment of this disclosure, the first time point and / or the second time point (and / or any other time point disclosed in this disclosure) are time points separated from a predetermined event by a predetermined time interval. The predetermined event may be part of and / or related to a dynamic inspection of the area to be inspected. The event may, for example, be the start of a dynamic inspection. The first time point and / or the second time point (and / or any other time point disclosed in this disclosure) may be within a predetermined time interval after the start of the dynamic inspection, such as within a specific number of seconds or minutes after the start of the dynamic inspection. The event may, for example, be the application of a contrast agent. The first time point and / or the second time point (and / or any other time point disclosed in this disclosure) may be within a predetermined time interval from the application of the contrast agent, such as within a predetermined number of seconds or minutes before or after the application of the contrast agent.
[0104] A second medical image is used as both input and target data. In other words, the conditional generative model is configured to reconstruct the second medical image. The reconstructed second medical image is a synthetic medical image.
[0105] The image embedding of the first medical image is used as a condition for the reconstruction of the second medical image.
[0106] In other words, the conditional generation model is configured and trained to generate a reconstructed second medical image representing the examination region of the examination object at a second time point, based on the image embedding of a first medical image representing the examination region of the examination object at a first time point as conditions. Generating the reconstructed second medical image typically involves inputting the second medical image and the image embeddings of the first medical image into the conditional generation model, and using the image embedding of the first medical image as conditions to reconstruct the synthetic medical image.
[0107] It is also possible that the conditional generation model is configured and trained to generate a reconstructed second medical image based on multiple first medical images (e.g., two, three, four, five, six, seven, or more than seven first medical images).
[0108] For example, the conditional input may include one or more image embeddings of two or more (e.g., three, four, five, six, seven or more) first medical images representing the examination area of the examined object at a first time point, and the target data may include a second medical image representing the examination area of the examined object at a second time point. In other words, the conditional generation model can be configured and trained to generate a reconstructed second medical image based on one or more image embeddings of multiple first medical images as conditions.
[0109] Multiple medical images may be, for example, medical images of different modalities and / or they may have been generated under different measurement conditions.
[0110] The term "modality" refers to a specific imaging technique or method used to generate medical images. Common modalities include X-ray, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, positron emission tomography (PET), and single-photon emission computed tomography (SPECT). Each modality offers unique advantages and is used to visualize different aspects of the body, such as bones, soft tissues, blood flow, or metabolic activity.
[0111] Different measurement conditions may involve, for example, energy, measurement sequence, contrast agent medium and / or similar factors.
[0112] In the case of MRI, medical images generated under different measurement conditions may be, for example, T1-weighted images, T2-weighted images, proton density images, diffusion-weighted images, images with and / or without contrast agents, images with different contrast agents, images with different amounts of contrast agents, and / or other / additional images.
[0113] In the case of computed tomography (CT) scans, medical images generated under different measurement conditions can be, for example, CT scans generated at different X-ray photon energies, CT scans with and / or without contrast agents, CT scans with different contrast agents, CT scans with different amounts of contrast agents, and / or other / additional images.
[0114] Alternatively, the conditional input may include one or more image embeddings of multiple medical images representing the examination area of the subject at different time points.
[0115] For example, it is possible that a conditional generation model is configured and trained to generate a reconstructed third medical image based on one or more image embeddings of one or more first medical images and one or more image embeddings of one or more second medical images, wherein the one or more first medical images represent the examination region of an object at a first time point, the one or more second medical images represent the examination region of an object at a second time point, and the (reconstructed) third medical image represents the examination region of the object at a third time point, wherein the first time point, the second time point, and the third time point are different time points.
[0116] Three time points t1, t2, and t3 can form any of the following time series: t1→t2→t3, t2→t3→t1, t3→t1→t2, t1→t3→t2, t2→t1→t3, t3→t2→t1. The sequence t1→t2→t3 means that the second time point is after the first time point, and the third time point is after the second time point. The intervals between consecutive time points can be the same or different.
[0117] For example, it is possible that a conditional generation model is configured and trained to generate a reconstructed fourth medical image based on one or more image embeddings of one or more first medical images, one or more image embeddings of one or more second medical images, and one or more image embeddings of one or more third medical images, wherein the one or more first medical images represent the examination region of the subject at a first time point, the one or more second medical images represent the examination region of the subject at a second time point, the one or more third medical images represent the examination region of the subject at a third time point, and the (reconstructed) fourth medical image represents the examination region of the subject at a fourth time point, wherein the first time point, the second time point, the third time point, and the fourth time point are different time points.
[0118] The four time points t1, t2, t3, and t4 can form any of the following time series: t j →t k →tl →t m Where j, k, l, and m can be the numbers 1, 2, 3, or 4, and j ≠ k ≠ l ≠ m. Sequence t j →t k →t l →t m This means that time point t k At time t j After that, at time t l At time t k After that, and at time t m At time t l Then, the intervals between consecutive time points can be the same or different.
[0119] Typically, a conditional generative model is configured and trained to generate reconstructed medical images based on one or more image embeddings of a number p medical images, wherein the medical images represent the examination regions of an object at q different time points, and the reconstructed medical images represent the examination regions of an object at time points different from the q time points, where p and q are integers greater than or equal to 1, and p ≥ q.
[0120] For example, p can be 1, 2, 3, 4, 5, 6, 7, 8, 9 or 10, q can be 1, 2, 3, 4, 5, 6, 7, 8, 9 or 10, and p ≥ q.
[0121] In one embodiment of this disclosure, a conditional generation model is configured and trained to generate a reconstructed medical image based on one or more image embeddings of a number p medical images, wherein the p medical images represent examination regions of an object at p different time points, and the reconstructed medical image represents examination regions of an object at time points different from the p different time points, where p is 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10.
[0122] Medical images (one or more image embeddings from said medical images are used as conditional inputs) may represent, for example, the examination area of an examinee at one or more time points before and / or after the application of a contrast agent, while the reconstructed medical image represents the examination area of an examinee at another time point before and / or after the application of said contrast agent.
[0123] In one embodiment of this disclosure, a conditional generation model is configured and trained to generate reconstructed MRI images based on one or more image embeddings of a number p MRI images, wherein each of the p MRI images represents the liver or a portion of the liver of an examined subject at one or more periods before or after the administration of a hepatobiliary contrast agent, and the reconstructed MRI image represents the liver or a portion of the liver of an examined subject at another period before or after the administration of the hepatobiliary contrast agent, wherein p is 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10.
[0124] The period may include or include the plain scan period, arterial period, portal venous period, transition period, and / or hepatobiliary period.
[0125] For example, one or more of the p-number of MRI images may represent the liver or a portion of the liver of the subject in the plain scan phase (i.e., without contrast agent), arterial phase, portal venous phase, and / or transition phase, and the reconstructed medical images may represent the liver or a portion of the liver of the subject in the hepatobiliary phase.
[0126] For example, one or more of the p-number of MRI images may represent the liver or a portion of the liver of the subject in the arterial phase, portal venous phase, transitional phase, and / or hepatobiliary phase, and the reconstructed medical images may represent the liver or a portion of the liver of the subject in the plain scan phase (i.e., without contrast agent).
[0127] As described, one or more image embeddings of at least one first medical image are used as conditions for the reconstruction of a second medical image. Therefore, an image embedding is generated from each first medical image.
[0128] "Image embedding" is a numerical representation of an image that captures its salient features.
[0129] Image embedding typically captures the meaning or semantics of medical images. It aims to encode the high-level information and concepts present in medical images, allowing machines to understand and reason about their content. For example, information about morphology, color, structure, and / or relationships between structures contained in a medical image can be aggregated into an image embedding.
[0130] Image embeddings can be vectors, matrices, tensors, or other numerical arrangements.
[0131] Image embeddings are generated using an image encoder. The image encoder can be part of a conditional generation model or it can be a discrete unit.
[0132] For example, an image embedding can be obtained by passing a medical image through a pre-trained machine learning model and then extracting the output of one layer of that machine learning model.
[0133] The machine learning model used to generate image embeddings can be, for example, or include (e.g., a pre-trained) convolutional neural network.
[0134] Convolutional Neural Networks (CNNs) are commonly used to generate image embeddings. These artificial neural networks consist of multiple layers that progressively extract features at different levels of abstraction, thus capturing both low-level details and higher-level semantic concepts. A CNN can be, for example, part of a classifier or an autoencoder.
[0135] In one embodiment of this disclosure, an encoder using an optional pre-trained autoencoder is used to generate the image embedding.
[0136] An autoencoder is a neural network architecture primarily used for unsupervised learning and dimensionality reduction. It is designed to learn a compressed representation of input data and then reconstruct the original data from that compressed representation (embedding). An autoencoder typically consists of two main components: an encoder and a decoder. The encoder receives the input data and maps it to a lower-dimensional latent space representation, also known as an embedding. The decoder then receives this embedding and reconstructs the original input data from it. The goal of an autoencoder is to minimize the reconstruction error, which encourages the model to learn a compressed representation that captures the most salient features of the input data.
[0137] Autoencoders are typically implemented as artificial neural networks, including convolutional neural networks (CNNs) to extract features from medical images that serve as input data. One example of such an autoencoder is U-Net (see, for example, O. Ronneberger et al.). U-net: Convolutional networks for biomedical images segmentation (International Conference on Medical Image Computing and Computer-Assisted Intervention, 234-241, Springer, 2015, DOI: 10.1007 / 978-3-319-24574-4_28). Other embodiments of autoencoders are sparse autoencoders, denoising autoencoders, variational autoencoders (VAEs), and generative adversarial networks (GANs). Autoencoders can be (pre-)trained on (unannotated) images. Images used for pre-training can be medical images, but they can also be other images or include other images.
[0138] Autoencoders can be (pre)trained using self-supervised learning methods, meaning they do not require labeled data for training.
[0139] The term "pre-trained" refers to a model that has been pre-trained on a large dataset and is available for multiple uses. Pre-training involves training a model on a task or dataset that is often different from the specific task the model will be used for later. The pre-training process involves exposing the model to a large amount of data and allowing it to learn general patterns and representations from that data. This enables the model to capture common features and structures that are useful across a variety of related tasks. The model is typically trained using unsupervised or self-supervised learning methods, where labels or annotations are automatically generated or require no human intervention. Once the pre-training phase is complete, the model's weights and parameters can be saved and made available. Other researchers or practitioners can then use this pre-trained model as a starting point for their own tasks. By leveraging the pre-trained model, they can benefit from the learned representations and potentially achieve better performance even with limited training data.
[0140] In another embodiment of this disclosure, an image embedding is generated using a pre-trained visual transformer.
[0141] Transformers are widely used in a variety of natural language processing tasks, including machine translation, text summarization, and sentiment analysis.
[0142] The core of the transformer model is the transformer architecture, which relies heavily on attention mechanisms to efficiently process sequential data. Unlike traditional recurrent neural networks (RNNs) or convolutional neural networks (CNNs), transformers do not employ recurrence or convolution operations. Instead, they use attention mechanisms to capture the contextual relationships between words or lexical units in a sequence.
[0143] Transformer architectures typically consist of two main components: an encoder and a decoder. The encoder processes the input sequence, modeling its contextual relationships, while the decoder generates the output sequence based on the encoded information. Both the encoder and decoder are usually composed of multi-layered attention mechanisms and feedforward neural networks. Attention mechanisms allow the model to focus on different parts of the input sequence while considering dependencies between words.
[0144] Transformers have significantly contributed to the advancement of machine learning, particularly in natural language processing tasks. Their ability to efficiently capture contextual information has achieved state-of-the-art performance on various benchmarks and has paved the way for numerous applications in the field (see, for example, T. Lin et al.). A survey of transformers AI Open, Volume 3, 2022, pp. 111-132.
[0145] Transformers have been applied to images (visual transformers).
[0146] In visual transformers, the input image is typically divided into a series of blocks, which are then flattened and fed into a series of transformer layers. These transformer layers include attention modules and feedforward neural networks. The attention mechanism allows the model to capture the relationships between different blocks and learn global contextual information, while the feedforward network implements non-linear transformations (see, for example, S. Khan et al.). Transformers in Vision : A Survey , arXiv:2101.01169v5).
[0147] The key advantage of visual transformers lies in their ability to model long-range dependencies and capture global context, which is crucial for understanding complex visual patterns and relationships.
[0148] Like autoencoders, visual transformers can be pre-trained. Visual transformers can be pre-trained using supervised, self-supervised, or unsupervised methods.
[0149] Visual transformers can be pre-trained using the DINO method. DINO (Self-Distillation Unlabeled) is a self-supervised learning method specifically designed to improve the performance of visual transformers on image classification tasks (see, for example, M. Caron et al.). Emerging Properties in Self-Supervised Vision Transformers , arXiv:2104.14294v2).
[0150] Self-supervised learning is a type of machine learning paradigm in which a model is trained to learn from the data itself, without the need for manually labeled annotations. Instead of relying on external labels provided by humans, the model generates its own supervisory signals from the input data, making it a form of unsupervised learning.
[0151] In traditional self-supervised learning, the model is trained on a pretext task, where labels are generated from the input data itself without requiring manual annotation. The model learns certain attributes or relationships in the prediction data, which in turn helps it learn meaningful representations. These representations can then be transferred to downstream tasks.
[0152] DINO introduces a novel approach to self-supervised learning for visual transformers by leveraging two main components: clustering and distillation. Initially, the model is trained to cluster augmented views of the input data. This clustering helps the model discover semantically similar instances within the dataset. Then, a distillation process is performed, where the model learns to transfer knowledge from a teacher network to a student network. The teacher network provides soft objectives or guidance to the student network, which helps improve the student's performance. By combining clustering and distillation, DINO enables the model to learn more robust and discriminative representations, resulting in better generalization and performance on downstream tasks such as image classification.
[0153] In another embodiment of this disclosure, the visual transformer is pre-trained using the DiNOv2 method. DiNOv2 (Discriminative Noise Contrast Learning V2) is another self-supervised method for training visual transformers (see, for example, M. Oquab et al.). DINOv2: Learning Robust Visual Features without Supervision ,arXiv:2304.07193v1).
[0154] In another embodiment of this disclosure, the image embedding is generated by an image encoder using a pre-trained CLIP model.
[0155] CLIP (Contrastive Language-Image Pretraining) is a framework in the field of machine learning that combines natural language processing and computer vision to understand and generate multimodal representations of images and text. CLIP encodes text and images into the same embedding space (see, for example, A. Radford et al.). Learning Transferable Visual Models From Natural Language Supervision, arXiv:2103.00020v1 ).
[0156] CLIP is pre-trained in a self-supervised manner, where it learns joint representations using a large-scale dataset of images and their associated text. The model is trained to associate images and their textual descriptions by maximizing the similarity in the learned embedding space. This allows CLIP to understand and reason about images and text in a shared semantic space. The base model uses a ViT-L / 14 transform architecture as the image encoder and a masked self-attention transform as the text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.
[0157] CLIP's key innovation lies in its ability to generalize across different domains and tasks. By training on a wide range of image-text pairs, CLIP can perform multiple tasks without requiring fine-tuning for specific tasks. For example, CLIP can perform zero-shot image classification, where it can classify images into categories never seen during training based solely on text descriptions.
[0158] In one embodiment of this disclosure, an image encoder using a pre-trained CLIP model, which is pre-trained on medical images, such as BiomedCLIP (see, for example, S. Zhang et al.): Large-Scale Domain-Specific Pretraining for Biomedical Vision-Language Processing arXiv:2303.00915v1).
[0159] However, the models and training procedures listed in this article are merely examples; image embeddings can also be generated in other ways.
[0160] It is also possible to generate more than one image embedding for medical images (e.g., two, three, four, or any other number).
[0161] Different image embeddings can be generated from medical images (e.g., using different embedding methods).
[0162] If multiple image embeddings are available, these image embeddings can be combined into a single image embedding, for example, through concatenation, average pooling, attention-weighted pooling, and / or other combination methods.
[0163] Image embeddings can be generated and saved before training the conditional generative model. Alternatively, image embeddings can be generated during the training of the conditional generative model.
[0164] Image embedding for generating medical images typically involves: inputting the medical image into an image encoder, and receiving the image embedding of the medical image as the output of the image encoder.
[0165] Training a conditional generative model typically involves the following steps: For each dataset: - The image embedding of a first medical image and a second medical image are input into a conditional generation model, wherein the image embedding of the first medical image is used as a condition in the generation of the reconstructed second medical image. - Receive the reconstructed second medical image as the output of the conditional generation model. - Determine the discrepancy between the second medical image and the reconstructed second medical image, for example, through a loss function. - Reduce bias by modifying the model parameters of the conditional generation model.
[0166] Training of the conditional generative model can be terminated when the stopping criteria are met. Such stopping criteria may be, for example: a predefined maximum number of training steps / cycles / rounds has been executed; modifying the model parameters can no longer reduce the deviation between the output data and the target data; a predefined minimum of the loss function has been reached; and / or an extreme value of another performance value has been reached (e.g., a maximum or minimum value).
[0167] The trained conditional generative model can be saved, transferred to a separate computer system, and / or used to generate synthetic medical images.
[0168] Training in Figure 1 and Figure 2 The examples are illustrated schematically.
[0169] Figure 1 An implementation scheme for training a conditional generative model is illustrated schematically.
[0170] In the first step, multiple datasets are received. For clarity, Figure 1 The image shows only one dataset, DS.
[0171] Each dataset DS represents the inspection area of the object being inspected.
[0172] exist Figure 1 In the illustrated embodiment, the subject of the examination is a human, and the examination area includes the human lungs.
[0173] It should be noted that, Figures 1 to 11 In this context, the human lung is selected as an example of the examination area for the subject of examination. The depicted human lung is merely a representation of any part of any subject being examined. In other words, Figures 1 to 11 The lungs of the human shown could also be another part of the subject of examination, such as the liver, kidneys, heart, lungs, brain, stomach, bladder, prostate, intestines, thyroid, eyes, breasts, or part of said sites, or another part of the body of a mammal (e.g., human).
[0174] Each dataset DS includes at least two medical images: a first medical image I1 and a second medical image I2.
[0175] The first medical image I1 represents the examination area of the subject at a first time point. The second medical image I2 represents the examination area of the subject at a second time point. The first and second time points are different time points.
[0176] A first medical image I1 is input to an image encoder IE. The image encoder IE generates an image embedding E based on the first medical image I1.
[0177] The second medical image I2 and its image embedding E are input into the conditional generation model CGM. The conditional generation model CGM is configured and trained to generate the reconstructed second medical image RI2 based on the image embedding E of the first medical image I1 and the second medical image I2. The image embedding E of the first medical image I1 is used as a condition for generating the reconstructed second medical image RI2.
[0178] Conditional generation model (CGM) includes the noisy model (NM) and the denoising model (DM).
[0179] The noisy model NM is configured to receive input data (i.e., the second medical image I2) and generate noisy data in response to receiving the input data. The noisy model introduces noise into the input data to blur the input data after multiple stages.
[0180] The denoising model DM is configured to reconstruct the input data (i.e., the second medical image I2) from the noisy data. The denoising model DM is configured to produce samples that match the input data after several stages.
[0181] It should be noted that the diffusion method can be performed in real space (e.g., pixel space, voxel space, or four-dimensional pixel space, depending on the case) or latent space.
[0182] The loss function LF was used to quantify the deviation between the second medical image I2 and the reconstructed second medical image RI2.
[0183] Bias can be reduced by modifying the model parameters of the conditional generation model (CGM).
[0184] Figure 1 The process shown is executed on multiple datasets until a stopping criterion is met.
[0185] If image embeddings are used as conditions in the reconstruction of medical images, a portion of the image embedding can be masked. Through masking, the conditional generative model is forced to compensate for missing information. For example, it learns to extract global information from local information.
[0186] The masked portion can be randomly selected or specifically chosen. The proportion of the masked portion can be constant or variable. Examples of masking are described in... Figure 4 , Figure 6 , Figure 7 and Figure 8 As shown in the image.
[0187] Figure 2 Another implementation scheme for training a conditional generative model is illustrated schematically.
[0188] Figure 2 The training shown is Figure 1 The difference in the training shown is that the image embeddings of several images are used as conditions for reconstruction.
[0189] In the first step, multiple datasets are received. For clarity, Figure 2 The image shows only one dataset, DS.
[0190] Each dataset DS represents the inspection area of the object being inspected.
[0191] exist Figure 2 In the embodiment shown, the subject of the examination is a human being, and the examination area includes the human lungs.
[0192] The dataset DS includes four medical images: the first medical image I1, the second medical image I2, the third medical image I3, and the fourth medical image I4.
[0193] The fourth medical image, I4, is the reconstructed image.
[0194] The first medical image I1, the second medical image I2, and the third medical image I3 differ from the fourth medical image I4 in the timing in which they represent the examination area.
[0195] It is possible that the first medical image I1, the second medical image I2, the third medical image I3, and the fourth medical image I4 represent the examination area at different time points. However, it is also possible that two or three of the first, second, and third medical images represent the examination area at the same time point, but in different modalities and / or under different measurement conditions.
[0196] In one embodiment of this disclosure, the first medical image I1, the second medical image I2, the third medical image I3, and the fourth medical image I4 represent the examination area at different time points.
[0197] Image embeddings are generated from each of the three medical images I1, I2, and I3 using an image encoder IE. A first image embedding E1 is generated from the first medical image I1; a second image embedding E2 is generated from the second medical image I2; and a third image embedding E3 is generated from the third medical image I3.
[0198] Medical image I4 is fed into a conditional generative model (CGM). The CGM generates a reconstructed medical image RI4. Embeddings E1, E2, and E3 are used as conditions in generating the reconstructed medical image RI4.
[0199] Image embeddings E1, E2, and E3 can be combined into a single embedding. This is the general case, and applies not only to [other applications]. Figure 2 The training shown is as follows: if several image embeddings can be used as conditions for reconstruction, they can be combined into a single embedding, i.e., a combined image embedding.
[0200] There are several options for combining multiple image embeddings into a single embedding.
[0201] Multiple image embeddings can be combined into a single embedding by concatenation, which involves end-to-end concatenation of the image embeddings. If the image embeddings are vectors, concatenation can create longer vectors or matrices. If the image embeddings are matrices, concatenation can create matrices or tensors with more rows or columns. This method (concatenation) preserves all the original information but may result in high-dimensional conditional inputs.
[0202] Multiple image embeddings can be combined into a single embedding by summation (i.e., by summing the image embeddings element by element).
[0203] Multiple image embeddings can be combined into a single embedding by performing principal component analysis (PCA) and generating embeddings based on the identified principal components.
[0204] Multiple image embeddings can be combined into one embedding by averaging (i.e., by taking the element-wise average of the image embeddings (e.g., the arithmetic mean)).
[0205] Multiple image embeddings can be combined into a single embedding by taking a weighted average. Taking a weighted average is similar to taking an average, but a weight is assigned to each image embedding and / or each dimension of the image embedding before averaging. The weights can be determined, for example, based on the importance of each image embedding and / or dimension. For example, the weights can be learned. It is possible that an image encoder or a downstream artificial neural network that combines the image embeddings is included in the training of the conditional generative model, and attention weights are learned during training.
[0206] It is possible that the image encoder (e.g., Figures 1, 2, 5, 6, 7, 9, 10 and / or Figure 11 The image encoder diagram shown is or includes a CNN, and the parameters of the CNN are learned during the training of the conditional generative model. For example, a CNN can perform one-dimensional convolutions on the elements of an image embedding, thereby merging the image embeddings into a single embedding.
[0207] Figures 3 to 8 A schematic embodiment of image embedding generation is shown, which can be used as a condition for reconstructing medical images. Figure 3 , Figure 4 and Figure 8 In the illustrated embodiment, the image encoder is based on a visual transformer. It should be noted that the image encoder of this disclosure is not limited to visual transformers.
[0208] Figure 3 A schematic example of how to generate image embeddings from medical images is shown.
[0209] The medical image I1 is divided into fixed-size blocks P1 to P9. Block embeddings are generated by flattening the blocks P1 to P9 and mapping them to dimensions corresponding to the input dimension of the transformer T via linear projection. a to i Position embeddings 1 through 9 are added to the block embedding to preserve positional information. The resulting sequence is used as input to transformer T.
[0210] In the illustrated embodiment, the sequence is preceded by an embedding at position 0. It can contain global information about the medical image I1, such as when and / or at what stage of the examination it was generated and / or which examination area it displays and / or the modality of the medical image and / or the measurement conditions that generated it.
[0211] It is possible to add other / further embeddings to the sequence. For example, one or more learnable input lexical units can be added, which the conditional generative model can use as a "register" (see, for example, T. Darcet et al.). Vision Transformers Need Registers , arXiv:2309.16588v1).
[0212] Transformer T generates image embedding E1 from this sequence.
[0213] This process of generating image embeddings is described, for example, in the following literature: A. Dosovitsky et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale ,arXiv:2010.11929v2).
[0214] Figure 4 A schematic embodiment of how multiple portions of an image embedding can be masked is shown. In the illustrated embodiment, blocks P1, P4, P5, and P8 are masked. The grayscale values of the masked blocks can be set to zero, for example. This also sets the corresponding image tokens of image embedding E1 to zero. However, masking does not have to be performed at the block level; it can also be performed at the image token level.
[0215] Figure 5 A schematic embodiment is shown that generates several image embeddings and combines these image embeddings into a single image embedding.
[0216] Figure 5 Three medical images, I1, I2, and I3, are shown. These medical images can be... Figure 2 As shown in the image and above regarding Figure 2 The three medical images described are I1, I2, and I3.
[0217] Blocks are generated from each of the three images and fed to the image encoder IE. The image encoder IE generates a first image embedding E1 based on blocks from the first medical image I1, a second image embedding E2 based on blocks from the second medical image, and a third image embedding E3 based on blocks from the third medical image I3. The image embeddings can be as follows: Figure 3 The described generation.
[0218] Image embeddings E1, E2, and E3 are combined into a single image embedding EC. This combined image embedding EC can be used as a condition when reconstructing medical images (e.g., in...). Figure 2 (When generating the reconstructed medical image RI4).
[0219] Image embeddings E1, E2, and E3 can be combined into a single image embedding EC in a variety of ways, such as by concatenation, by summing the image embeddings element-wise together, by performing principal component analysis (PCA) and generating embeddings based on principal components, by taking the element-wise average of the image embeddings (e.g., arithmetic mean), by taking the element-wise maximum of the image embeddings (e.g., arithmetic mean), by taking a weighted average, and / or by using a trainable machine learning model (such as an artificial neural network).
[0220] In the case of multiple image embeddings, blocks and / or image terms can also be masked. Figure 6 An example of masking blocks and / or image terms is shown. For example... Figure 6 As shown, for example, blocks representing the same sub-regions of the inspection area can be masked randomly or according to defined rules. When image embeddings E1, E2, and E3 are combined by summation, average pooling, or max pooling to form a combined image embedding EC, the corresponding image terms of the combined image embedding EC will also take a value of zero.
[0221] Figure 7 Another embodiment of masking blocks and / or image terms is shown. Figure 7 In the illustrated embodiment, the individual medical image is masked, in this case, medical image I2. If the medical image represents a period in a dynamic examination, information about that period is missing when reconstructing the medical image. The conditional generation model is forced to compensate for the missing information from images representing other periods. This makes the conditional generation model invariant to the number of periods received during inference. This also applies similarly to all other different features that a medical image can represent (e.g., modalities and / or measurement conditions).
[0222] Figure 8 Another embodiment of generating image embeddings based on multiple medical images is shown. Figure 8 In the illustrated embodiment, medical images I1, I2, and I3 are divided into blocks. This generates three sets of blocks. A defining portion of each set is used to create an image embedding EC such that each sub-region of the examination area is represented once by a block. In other words, a new image I is generated from the blocks of images I1, I2, and I3, which is composed of the blocks of images I1, I2, and I3 in such a manner that each sub-region of the examination area is represented once by a block. In this embodiment, the new image I consists of blocks P11, P22, P31, P43, P52, P61, P73, P83, and P92, where the first digit indicates the block's position, and the second digit indicates the number of its source image.
[0223] The proportion of each medical image block and / or which blocks from which medical images were used to generate the new image can vary in each training cycle.
[0224] and Figure 7 The method is the same. Figure 8 The method also results in the conditional generative model being invariant to the number of periods / modalities and the number of periods / modalities.
[0225] As described, trained conditional generative models can be used to generate synthetic medical images.
[0226] The use of a trained model is also known as inference.
[0227] In the first step, at least one medical image of the new subject of examination is received.
[0228] The term "new" typically means that data from new examinees was not used to train the conditional generative model. However, it is possible that data from new examinees was used to train the conditional generative model. New examinees are usually of the same type as the examinees representing the training data. For example, if a conditional generative model is trained using training data representing humans, then the new examinees are typically also humans.
[0229] The term "receive" includes retrieving one or more medical images and receiving one or more medical images, for example, transmitted to a computer system containing this disclosure. The one or more medical images may be received from an MRI scanner, a CT scanner, or any other device used to generate medical images (as applicable). The one or more medical images may be read from one or more data storage devices.
[0230] The at least one medical image represents an examination region of a new subject at a first time point. This examination region is typically the same as, or at least a part of, the examination region used during training, or includes the examination region. This first time point typically corresponds to the first time point during model training.
[0231] Based on the at least one medical image, an image encoder is used to generate at least one image embedding. Inference typically uses the same image encoder as the training image.
[0232] Synthetic medical images are generated using a trained conditional generation model based on at least one image embedding as a condition.
[0233] For example, if the trained conditional generation model is a conditional diffusion model with both a denoising model and a denoising model, the denoising model can be discarded (in other words, the trained conditional generation model includes a denoising model but does not need to include a denoising model), and noisy data can be input into the denoising model. Then, the denoising model uses the at least one image embedding as a condition to progressively generate synthetic medical images from the noisy data.
[0234] The synthetic medical image represents the examination area of a new subject at a second time point. This second time point corresponds to the corresponding time point in the training data, as in the first time point.
[0235] Synthetic medical images can be output (e.g., displayed on a monitor or printed using a printing device) and / or stored in a data storage device and / or transmitted to a separate computer system.
[0236] Figures 9 to 11 The generation of synthetic medical images is illustrated schematically and by way of example.
[0237] Figure 9 An implementation scheme for generating synthetic medical images based on medical images is shown.
[0238] Using a trained conditional generation model (CGM) t To generate synthetic medical images (SI). The trained conditional generation model (CGM) is used. t It can be as about Figure 1 The described training has been performed.
[0239] Receive medical images of new examination subjects I n The superscript n indicates that the image originated from a new examination subject. Medical Image I n This indicates the examination area of a new object at the first point in time during a dynamic examination. (Medical Image I) n It is fed into the image encoder IE. The image encoder IE is based on medical image I. n To generate image embeddings E n Noisy data (ND) is provided. The noisy data (ND) is fed into a trained conditional generation model (CGM). t The denoising model DM. Similarly, image embedding E n The data is fed into the trained conditional generation model (CGM). t The trained conditional generative model (CGM) t Based on noisy data ND and based on image embedding E as a condition n This is used to generate a synthetic medical image (SI). The synthetic medical image (SI) represents the examination area of a new subject at a second time point. The first time point and the second time point are different time points. The first time point and the second time point correspond to the corresponding time points in the training data.
[0240] Figure 10 An implementation scheme for generating synthetic medical images based on several medical images is shown.
[0241] Using a trained conditional generation model (CGM) t To generate synthetic medical images (SI). The trained conditional generation model (CGM) is used. t It can be as about Figure 2 The described training has been performed.
[0242] exist Figure 10 In the illustrated embodiment, three medical images of the new examination subject are received: the first medical image I1 n Second Medical Image I2 n With third medical image I2 n .
[0243] It is possible that the first medical image I1 n Second Medical Image I2 n With the third medical image I3 n This refers to the examination area of a new subject at different points in time. However, it is also possible for one or more medical images to represent the examination area at the same point in time, but in different modalities and / or under different measurement conditions.
[0244] In one embodiment of this disclosure, the first medical image I1 n Second Medical Image I2 n With the third medical image I3 n This indicates the inspection area at different points in time.
[0245] Image embeddings are generated from each medical image using an image encoder (IE). Based on the first medical image I1... n To generate the first image embedding E1 n Based on the second medical image I2 n To generate a second image embedding E2 n And based on third medical image I3 n To generate a third image embedding E3 n .
[0246] Image embedding E1 n E2 n and E3 n The combined image is embedded in EC n Combinations can be used to generate models with training conditions, and as per [the relevant information]... Figure 2 and Figure 5 It is executed in the same manner as described.
[0247] Combined image embedding EC nIt is used as a condition for generating synthetic medical images (SI).
[0248] Noisy data (ND) is provided. The noisy data (ND) is fed into the trained conditional generation model (CGM). t The denoising model DM. Similarly, the combined image embedding EC n The data is fed into the trained conditional generation model (CGM). t The trained conditional generative model (CGM) t Based on noisy data and based on combined image embedding EC n Synthetic medical image SI is generated as a condition. Synthetic medical image SI represents the examination area of a new subject at a given time point, which corresponds to medical image I1. n I2 n and I3 n The time points representing the inspection areas are different.
[0249] If there are different ways to combine multiple image embeddings into a combined image embedding during training of the conditional generative model, multiple combined image embeddings can be generated during inference, and different synthetic medical images can be generated based on each combined image embedding. The different synthetic medical images generated in this way can be combined into a single synthetic medical image, for example, by element-wise averaging or other methods. This also applies similarly to... Figure 8 The scenario shown illustrates how a new medical image I is generated based on medical images I1, I2, and I3. This is in... Figure 11 The example is shown schematically.
[0250] Figure 11 Another implementation scheme for generating synthetic medical images based on several medical images is shown.
[0251] In the first step, three medical images I1 are received. n I2 n and I3 n Each medical image represents the examination area of the new subject.
[0252] It is possible that the first medical image I1 n Second Medical Image I2 n With the third medical image I3 n This refers to the examination area of a new subject at different points in time. However, it is also possible that one or more medical images represent the examination area at the same point in time, but in different modalities and / or under different measurement conditions. In one embodiment of this disclosure, the first medical image I1 n Second Medical Image I2 n With the third medical image I3 nThis indicates the inspection area at different points in time.
[0253] Each medical image is divided into multiple blocks.
[0254] There are multiple ways to combine three images I1 n I2 n With I3 n The blocks are combined to form a new image. Figure 11 The illustrated embodiment shows two possibilities: generating two new images IN1 and IN2 based on blocks. However, it is also possible to generate more than two new images, such as three, four, or more than four.
[0255] The new images IN1 and IN2 have the property that each sub-region of the examined region is determined by image IN1. n I2 n and I3 n It is represented by a block of an image.
[0256] The first image embedding E1 is generated using the image encoder IE and based on the first new image IN1. n The second image embedding E2 is generated using the image encoder IE and based on the second new image IN2. n .
[0257] First noisy data ND1 is provided. The first noisy data ND1 is fed into the trained conditional generation model CGM. t The denoising model DM. The trained conditional generative model CGM. t E1 is embedded using the first image. n As a condition, and based on the first noisy data ND1, a first synthetic medical image SI1 is generated.
[0258] Similarly, a second noisy data point, ND2, is provided. This second noisy data point, ND2, is fed into the trained conditional generation model, CGM. t The denoising model DM. The trained conditional generative model CGM. t Embed E2 using the second image n As a condition, and based on the second noisy data ND2, a second synthetic medical image SI2 is generated.
[0259] Synthetic medical images SI1 and SI2 represent the examination area of a new subject at a given time point, which is related to medical image I1. n I2 n With I3 n The time points indicated for the inspected areas are different.
[0260] Synthetic medical images SI1 and SI2 are combined to obtain a combined synthetic medical image SIC.
[0261] The combined composite image SIC can be generated, for example, from the first composite image SI1 and the second composite image SI2 by averaging element by element.
[0262] Figure 12 An implementation scheme of a computer-based method of the present disclosure is illustrated in the form of a flowchart.
[0263] The method (100) includes the following steps: (110) Provide multiple datasets for multiple examination subjects, each dataset including at least two medical images: a first medical image and a second medical image, wherein the first medical image represents the examination area of the examination subject at a first time point, and the second medical image represents the examination area of the examination subject at a second time point, wherein the first time point and the second time point are different time points. (120) Provides an image encoder, wherein the image encoder is configured to generate an image embedding based on a medical image. (130) For each dataset: use the image encoder to generate an image embedding of the first medical image, (140) Provides a conditional generation model, wherein the conditional generation model is configured to generate a reconstructed medical image based on a medical image, conditions, and model parameters. (150) Training the conditional generation model, wherein for each dataset, the training includes: (151) A reconstructed second medical image is generated based on the image embedding of the first medical image and the second medical image, wherein the image embedding of the first medical image is used as a condition in the generation of the reconstructed second medical image. (152) Determine the deviation between the second medical image and the reconstructed second medical image. (153) The bias is reduced by modifying the model parameters of the conditional generation model. (160) Receive at least one medical image of a new subject, wherein the at least one medical image represents the examination area of the new subject at a first time point. (170) Generating at least one image embedding based on at least one medical image of a new examination subject, wherein generating at least one image embedding comprises: inputting the at least one medical image into an image encoder, (180) Using the at least one image embedding as conditional input to a trained conditional generation model to generate synthetic medical images. (190) Output and / or store the synthetic medical image, and / or transmit the synthetic medical image to a separate computer system.
[0264] The computer-implemented method of this disclosure can be divided into a training phase and an inference phase.
[0265] exist Figure 12 In the embodiment shown, the training phase TP includes steps (110) to (153), and the inference phase IP includes steps (160) to (190).
[0266] The operations taught herein can be performed by at least one computer system specially constructed for the desired purpose or a general-purpose computer system specially configured for the desired purpose, by at least one computer program stored in a typical non-transitory computer-readable storage medium.
[0267] A "computer system" is an electronic data processing system that processes data through programmable computational rules. Such a system typically includes a "computer" and peripheral devices, wherein the "computer" is a unit that includes a processor for performing logical operations.
[0268] In computer technology, "peripheral devices" refer to all devices connected to a computer and used to control the computer and / or as input and output devices. Examples of peripheral devices include monitors (screens), printers, scanners, mice, keyboards, drives, cameras, microphones, speakers, etc. Internal ports and expansion cards are also considered peripheral devices in computer technology.
[0269] Modern computer systems are generally classified as desktop PCs, portable PCs, laptops, notebooks, netbooks, tablet PCs, and so-called handheld devices (e.g., smartphones); all of these systems can be used to implement the methods disclosed herein.
[0270] The term "non-transient" is used herein to exclude temporary, propagating signals or waves, but in addition to include any volatile or non-volatile computer storage technology applicable to this application.
[0271] The term “computer” should be broadly interpreted to encompass any type of electronic device capable of data processing, including, by way of non-limiting example, personal computers, servers, embedded cores, computing systems, communication devices, processors (e.g., digital signal processors (DSPs), microcontrollers, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc.) and other electronic computing devices.
[0272] As used above, the term "processing" is intended to include any type of computation, operation, or transformation of data represented by physical (e.g., electronic) phenomena that may occur or exist in, for example, the registers and / or memory of at least one computer or processor. The term "processor" includes a single processing unit or multiple distributed or remote such units.
[0273] Figure 13 A computer system (1) according to some example embodiments of the present disclosure is illustrated in more detail.
[0274] Generally, a computer system in an exemplary embodiment of this disclosure may be referred to as a computer and may include, be incorporated into, or be embodied in one or more fixed or portable electronic devices. A computer may include one or more of each of several components, such as a processing unit (20) connected, for example, to a memory (50) (e.g., a storage device).
[0275] The processing unit (20) may consist of one or more processors alone, or a combination of one or more processors and one or more memories. The processing unit (20) is typically any computer hardware capable of processing information, such as, for example, data, computer programs, and / or other suitable electronic information. The processing unit (20) consists of a set of electronic circuits, some of which may be packaged as integrated circuits or multiple interconnected integrated circuits (integrated circuits are sometimes more commonly referred to as “chips”). The processing unit (20) may be configured to execute computer programs, which may be stored locally on the processing unit (20) or otherwise stored in the memory (50) of the same or another computer.
[0276] Depending on the specific implementation, the processing unit (20) may be multiple processors, a multi-core processor, or some other type of processor. For example, it may be a central processing unit (CPU), a field-programmable gate array (FPGA), a graphics processing unit (GPU), and / or a tensor processing unit (TPU). Furthermore, the processing unit (20) may be implemented using several heterogeneous processor systems, where a main processor and one or more auxiliary processors reside on a single chip. As another exemplary embodiment, the processing unit (20) may be a symmetric multiprocessor system comprising multiple processors of the same type. In yet another embodiment, the processing unit (20) may be embodied as or otherwise include one or more ASICs, FPGAs, etc. Thus, while the processing unit (20) is capable of executing a computer program to perform one or more functions, the processing unit (20) in various embodiments may be capable of performing one or more functions without the aid of a computer program. In any case, the processing unit (20) may be suitably programmed to perform the functions or operations according to the exemplary embodiments of this disclosure.
[0277] Memory (50) is generally any computer hardware capable of storing information (such as, for example, data, computer programs (e.g., computer-readable program code (60)) and / or other suitable information) in a temporary and / or permanent manner. Memory (50) may include volatile and / or non-volatile memory and may be fixed or removable. Suitable embodiments of memory include random access memory (RAM), read-only memory (ROM), hard disk drive, flash memory, USB flash drive, removable computer floppy disk, optical disk, magnetic tape, or some combination thereof. Optical disk may include optical disc read-only memory (CD-ROM), optical disc read / write (CD-R / W), DVD, Blu-ray disc, etc. In many cases, memory may be referred to as computer-readable storage medium or data storage. Computer-readable storage medium is a non-transitory device capable of storing information and is distinguishable from computer-readable transmission medium (such as an electronic transient signal capable of carrying information from one location to another). The computer-readable medium described herein may generally refer to a computer-readable storage medium or a computer-readable transmission medium.
[0278] In addition to the memory (50), the processing unit (20) may also be connected to one or more interfaces for displaying, transmitting, and / or receiving information. The interfaces may include one or more communication interfaces and / or one or more user interfaces. The communication interfaces may be configured to transmit and / or receive information, such as transmitting information to other computers, networks, databases, etc., and / or transmitting information from other computers, networks, databases, etc. The communication interfaces may be configured to transmit and / or receive information via physical (wired) and / or wireless communication links. The communication interfaces may include interfaces (41) for connecting to a network, such as using technologies such as cellular phones, Wi-Fi, satellite, cable, digital subscriber line (DSL), fiber optics, etc. In some embodiments, the communication interfaces may include one or more short-range communication interfaces (42) configured to connect devices using short-range communication technologies such as NFC, RFID, Bluetooth, Bluetooth LE, ZigBee, infrared (e.g., IrDA), etc.
[0279] The user interface may include a display (30). The display (screen) may be configured to present or otherwise display information to a user, and suitable embodiments include liquid crystal displays (LCDs), light-emitting diode displays (LEDs), plasma display panels (PDPs), etc. The user input interface (11) may be wired or wireless and may be configured to receive information from a user into the computer system (1), such as for processing, storage, and / or display. Suitable embodiments of the user input interface include microphones, image or video capture devices, keyboards or keypads, joysticks, touch-sensitive surfaces (separate from or integrated into the touchscreen), etc. In some embodiments, the user interface may include automatic identification and data capture (AIDC) technology (12) for machine-readable information. This may include barcodes, radio frequency identification (RFID), magnetic stripes, optical character recognition (OCR), integrated circuit cards (ICCs), etc. The user interface may further include one or more interfaces for communicating with peripheral devices, such as printers, etc.
[0280] As indicated above, program code instructions (60) may be stored in memory (50) and executed by processing unit (20), which is thus programmed to perform the functions of the systems, subsystems, tools, and their respective elements described herein. It will be understood that any suitable program code instructions (60) may be loaded from a computer-readable storage medium onto a computer or other programmable device to produce a particular machine, such that particular machine becomes a means of performing the functions specified herein. These program code instructions (60) may also be stored in a computer-readable storage medium that can direct a computer, processing unit, or other programmable device to operate in a particular manner to produce a particular machine or article of manufacture. Instructions stored in a computer-readable storage medium can produce an article of manufacture, which becomes a means of performing the functions described herein. Program code instructions (60) may be retrieved from a computer-readable storage medium and loaded onto a computer, processing unit, or other programmable device to configure the computer, processing unit, or other programmable device to perform operations to be performed on or by the computer, processing unit, or other programmable device.
[0281] The retrieval, loading, and execution of program code instructions (60) can be performed sequentially, such that one instruction is retrieved, loaded, and executed at a time. In some example implementations, the retrieval, loading, and / or execution can be performed in parallel, such that multiple instructions are retrieved, loaded, and / or executed together. Execution of program code instructions (60) can produce computer-implemented processes that provide operations for carrying out the functions described herein, executed by a computer, processing circuitry, or other programmable means.
[0282] A combination of operations that support the execution of a specified function, either by executing instructions by a processing unit or by storing instructions in a computer-readable storage medium. Thus, a computer system (1) may include a processing unit (20) and a computer-readable storage medium or memory (50) coupled to processing circuitry, wherein the processing circuitry is configured to execute computer-readable program code instructions (60) stored in the memory (50). It will also be understood that one or more functions, and combinations thereof, may be implemented by a computer system and / or processing circuitry based on dedicated hardware that performs the specified function, or by a combination of dedicated hardware and program code instructions.
[0283] The computer system disclosed herein may be in the form of a laptop, notebook, netbook, and / or tablet PC; it may also be a component of an MRI scanner, CT scanner, ultrasound diagnostic instrument, or any other device used to generate and / or process medical images.
[0284] On the other hand, this disclosure provides a computer program product. Such a computer program product includes a non-volatile data carrier, such as a CD, DVD, USB flash drive, or other media for storing data. The computer program is stored on the data carrier. The computer program can be loaded into the working memory of a computer system (particularly into the working memory of the computer system described in this disclosure), wherein it can cause the computer system to perform the following steps: - Provides multiple datasets for multiple examination subjects, each dataset including at least two medical images: a first medical image and a second medical image, wherein the first medical image represents the examination area of the examination subject at a first time point, and the second medical image represents the examination area of the examination subject at a second time point, wherein the first time point and the second time point are different time points. - Provides an image encoder, wherein the image encoder is configured to generate image embeddings based on medical images. - For each dataset: use the image encoder to generate an image embedding of the first medical image. - Provides a conditional generation model, wherein the conditional generation model is configured to generate a reconstructed medical image based on a medical image, conditions, and model parameters. - Train the conditional generation model, wherein for each dataset, the training includes: A reconstructed second medical image is generated based on the image embedding of a first medical image and a second medical image, wherein the image embedding of the first medical image is used as a condition in the generation of the reconstructed second medical image. Determine the deviation between the second medical image and the reconstructed second medical image. The bias can be reduced by modifying the model parameters of the conditional generation model. - Receive at least one medical image of a new subject being examined, wherein the at least one medical image represents the examination area of the new subject being examined at a first time point. - Generate at least one image embedding based on at least one medical image of the new examination subject, wherein generating the at least one image embedding includes: inputting the at least one medical image into an image encoder. - Use the at least one image embedding as conditional input to a trained conditional generation model to generate synthetic medical images. - Output and / or store the synthetic medical images, and / or transmit the synthetic medical images to a separate computer system.
[0285] Computer programs can also be sold in combination with contrast agents. Such combinations are also referred to as kits. Such kits include both the contrast agent and the computer program. Alternatively, such kits may include the contrast agent and a device that allows the purchaser to obtain the computer program, such as downloading it from a website. These devices may include a link to the address of a website from which the computer program can be downloaded to an internet-connected computer system. Such devices may include a code (e.g., an alphanumeric string or QR code, or a DataMatrix code or barcode, or other optically and / or electronically readable code) that allows the purchaser to access the computer program. Such links and / or codes may, for example, be printed on the contrast agent packaging and / or on the contrast agent's instruction manual. Therefore, a kit is a combination product comprising a contrast agent and a computer program (e.g., in the form of access to the computer program or in the form of executable program code on a data carrier), which is offered for sale together.
[0286] In another aspect, this disclosure relates to the use of contrast agents in dynamic examination of the examination area of a subject.
[0287] The dynamic checks include: - Receive one or more medical images, wherein the one or more medical images represent the examination area of the subject at one or more time points before and / or after the application of contrast agent. - Provide an image encoder, wherein the image encoder is configured to generate an image embedding based on the one or more medical images. - Use the image encoder and generate at least one image embedding based on the one or more medical images. - Provides a trained conditional generation model, wherein the trained conditional generation model is trained to generate synthetic medical images based on image embeddings as conditions. - Generate synthetic medical images using the at least one image embedding as conditional input to the trained conditional generation model. - Output and / or store the synthetic medical images and / or transmit the synthetic medical images to a separate computer system. The training of the trained conditional generation model includes: - Provides multiple datasets of multiple reference examination subjects, each dataset including at least two reference medical images: one or more first reference medical images and a second reference medical image, wherein the one or more first reference medical images represent the examination area of the reference examination subject at one or more time points before and / or after the application of contrast agent, and the second reference medical image represents the examination area of the reference examination subject at another time point before and / or after the application of contrast agent. - For each dataset: use the image encoder to generate one or more reference image embeddings of the one or more first reference medical images. - Train the conditional generation model, wherein for each dataset, the training includes: A reconstructed second reference medical image is generated based on one or more reference image embeddings of the one or more first reference medical images and the second reference medical image, wherein the one or more reference image embeddings of the one or more first reference medical images are used as conditions in the generation of the reconstructed second reference medical image. Determine the deviation between the second reference medical image and the reconstructed second reference medical image. The bias is reduced by modifying the model parameters of the conditional generation model.
[0288] The term "reference" is used in some places in this disclosure to distinguish between data used for training and / or validating the conditional generation model and data used to generate synthetic medical images using the trained conditional generation model. Thus, data used for training and / or validating the conditional generation model refers to a "reference examination object," while data used to generate synthetic medical images refers to a (new) examination object. Similarly, medical images and image embeddings representing reference examination objects are referred to in some places as "reference medical images." However, the term "reference" should not be interpreted in any other restrictive sense; this distinction is used solely to prevent clarity objections in patent grant proceedings.
[0289] In one embodiment of this disclosure, each (reference) medical image is an MRI image.
[0290] In one embodiment of this disclosure, the dynamic examination is a dynamic MRI examination.
[0291] In one embodiment of this disclosure, the contrast agent is a hepatobiliary contrast agent.
[0292] In one embodiment of this disclosure, the examination area is a human liver or a portion thereof.
[0293] In one embodiment of this disclosure, the one or more medical images are one or more MRI images representing one or more of the following phases during a dynamic MRI examination of the liver or a portion thereof using a hepatobiliary contrast agent: plain phase, arterial phase, portal venous phase, and transition phase, and the synthetic medical image is a synthetic MRI image representing the liver or a portion thereof of the subject during the hepatobiliary phase.
[0294] In one embodiment of this disclosure, the conditional generation model is trained on multiple datasets of multiple reference subjects, each dataset including at least two reference MRI images: one or more first reference MRI images and a second reference MRI image, wherein the one or more first reference MRI images represent the liver or a portion thereof of the reference subject during one or more of the following phases of a dynamic MRI examination using hepatobiliary contrast agents: plain phase, arterial phase, portal venous phase, and transition phase, and the second reference MRI image represents the liver or a portion thereof of the reference subject during the hepatobiliary phase.
[0295] In one embodiment of the present invention, the computer-implemented method and / or the dynamic check include: - Receive one or more MRI images, wherein the one or more MRI images represent the liver or a portion thereof of the examined subject at one or more of the following periods before and / or after the application of hepatobiliary contrast agents: plain scan phase, arterial phase, portal venous phase, transition phase, - Provide an image encoder, wherein the image encoder is configured to generate an image embedding based on the one or more MRI images. - Use the image encoder and generate at least one image embedding based on the one or more MRI images. - Provides a trained conditional generation model, wherein the trained conditional generation model is trained to generate synthetic medical images based on image embeddings as conditions. - Synthetic MRI images are generated using the at least one image embedding as conditional input to the trained conditional generation model, wherein the synthetic MRI images represent the liver or a portion thereof of the examined subject during the hepatobiliary phase following the application of a hepatobiliary contrast agent. - Output and / or store the synthesized MRI images and / or transfer the synthesized MRI images to a separate computer system. The training of the trained conditional generation model includes: - Provides multiple datasets for multiple reference examination objects, each dataset including at least two reference MRI images: one or more first reference MRI images and a second reference MRI image. The one or more first reference MRI images represent the liver or a portion thereof of a reference subject during one or more of the following periods before and / or after the application of hepatobiliary contrast agents: plain scan, arterial phase, portal venous phase, transition phase, The second reference MRI image represents the liver or a portion thereof of the reference subject during the hepatobiliary phase following the application of hepatobiliary contrast agents. - For each dataset: use the image encoder to generate one or more reference image embeddings of the one or more first reference MRI images. - Train the conditional generation model, wherein for each dataset, the training includes: A reconstructed second reference MRI image is generated based on one or more reference image embeddings of the one or more first reference MRI images and the second reference MRI image, wherein the one or more reference image embeddings of the one or more first reference MRI images are used as conditions in the generation of the reconstructed second reference MRI image. Determine the deviation between the second reference MRI image and the reconstructed second reference MRI image. The bias is reduced by modifying the model parameters generated from the conditions.
Claims
1. A computer-implemented method, comprising: - Provide an image encoder (IE) configured to generate image embeddings (E) based on medical images. - Provides a trained Conditional Generative Model (CGM) t The trained conditional generation model (CGM) mentioned above. t The model is trained to generate reconstructed medical images based on medical images, conditions, and model parameters, wherein the training includes: Multiple datasets are provided for multiple examination subjects, each dataset (DS) including at least two medical images: a first medical image (I1) and a second medical image (I2), wherein the first medical image (I1) represents the examination area of the examination subject at a first time point, and the second medical image (I2) represents the examination area of the examination subject at a second time point, wherein the first time point and the second time point are different time points. For each dataset (DS): The image encoder (IE) is used to generate the image embedding (E) of the first medical image (I1). Using the conditional generation model (CGM), a reconstructed second medical image (RI2) is generated based on the image embedding (E) of the first medical image (I1) and the second medical image (I2), wherein the image embedding (E) of the first medical image (I1) is used as a condition in the generation of the reconstructed second medical image (RI2). Determine the deviation between the second medical image (I2) and the reconstructed second medical image (RI2). The bias can be reduced by modifying the model parameters of the conditional generative model (CGM). - Receive at least one medical image (I) of the new patient being examined. n ), wherein at least one medical image (I n () indicates the inspection area of the new inspection object at the first time point. - Using the image encoder (IE), based on the at least one medical image (I) of the new examination object. n To generate at least one image embedding (E) n ), - Using the at least one of the image embeddings (E n ) as a trained conditional generative model (CGM) t The conditions of input are used to generate synthetic medical images (SI). - Output and / or store the synthetic medical image (SI), and / or transmit the synthetic medical image (S) to a separate computer system.
2. The method of claim 1, wherein each of the plurality of examination subjects is a human being, wherein the new examination subject is a human being, and the examination area of each of the plurality of examination subjects and the examination area of the new examination subject is or includes a liver, kidney, heart, lung, brain, stomach, bladder, prostate, intestine, thyroid, eye, breast, or a part of said site or another part of the corresponding examination subject's body.
3. The method according to claim 1 or 2, wherein the at least two medical images (I1, I2) of each dataset (DS) are MRI images and / or CT images, and the at least one medical image (I1, I2) of the new examination subject is... n () is an MRI image or a CT image.
4. The method according to any one of claims 1 to 3, wherein the at least two medical images (I1, I2) of each dataset (DS) are MRI images, and the at least one medical image (I1, I2) of the new examination subject is... n () is an MRI image.
5. The method according to any one of claims 1 to 4, Each dataset (DS) includes p first medical images (I1, I2, I3) and a second medical image (I4). The first medical image (I1, I2, I3) represents the examination area of the subject at q different time points, and the second medical image (I4) represents the examination area of the subject at a time point different from the q time points, where p is 1, 2, 3, 4, 5, 6, 7, 8, 9 or 10, q is 1, 2, 3, 4, 5, 6, 7, 8, 9 or 10, and p ≥ q.
6. The method according to any one of claims 1 to 5, Each dataset (DS) includes at least two medical images comprising multiple first medical images (I1, I2, I3) and a second medical image (I4). Each of the plurality of first medical images (I1, I2, I3) represents the examination area of the object at different time points. The image embedding of the first medical image generated using the image encoder (IE) includes: Generate an image embedding (E1, E2, E3) for each of the plurality of first medical images (I1, I2, I3), and combine the image embeddings (E1, E2, E3) into a combined image embedding (EC). The generation of the reconstructed second medical image based on the image embedding of the first medical image and the second medical image includes: generating the reconstructed second medical image (RI4) based on the combined image embedding (EC) of the first medical image (I1, I2, I3) and the second medical image (I4), wherein the combined image embedding (EC) of the first medical image (I1, I2, I3) is used as a condition in the generation of the reconstructed second medical image (RI4).
7. The method according to any one of claims 1 to 6, At least one medical image received from a new patient includes: Receive multiple medical images (I1) of the new examination subject n I2 n , I3 n ), The multiple medical images (I1) n I2 n , I3 n At least some of the terms in the table represent the inspection areas of the new inspection object at different points in time. Generating at least one image embedding using the image encoder and based on at least one medical image of the new examination object includes: generating the plurality of medical images (I1) using the image encoder (IE). n I2 n , I3 n Image embeddings (E1) for each medical image in ) n E2 n E3 n ), and embed the image (E1) n E2 n E3 n ) combined into combined image embeddings (EC1) n ), Generating the synthetic medical image by using the at least one image embedding as conditional input to the trained conditional generation model includes: using the combined image embedding (EC1) n ) as the trained conditional generative model (CGM) t The synthetic medical image (SI) is generated by taking the conditional input of the input.
8. The method according to claim 7, Each dataset (DS) includes at least two medical images comprising multiple first medical images (I1, I2, I3) and a second medical image (I4). Each of the plurality of first medical images (I1, I2, I3) represents a portion of the human liver during one of the following phases of a dynamic contrast-enhanced MRI examination using a hepatobiliary contrast agent: plain phase, arterial phase, portal venous phase, transition phase, The second medical image (I4) represents a human liver or a portion thereof during the hepatobiliary phase of the dynamic contrast-enhanced MRI examination. The multiple medical images (I1) of the new examination object n I2 n , I3 n Each medical image in the image represents the liver or a portion thereof of the newly examined subject during one of the following phases of a dynamic contrast-enhanced MRI examination using hepatobiliary contrast agents: plain phase, arterial phase, portal venous phase, transition phase, The synthetic medical image (SI) therein represents the liver or a portion thereof of the new subject during the hepatobiliary phase of the dynamic contrast-enhanced MRI examination.
9. The method according to any one of claims 1 to 8, The conditional generation model (CGM) mentioned above is a conditional diffusion model, which includes a noisy model (NM) and a denoising model (DM). The noise-adding model (NM) is configured to receive the second medical image (I2) and generate noisy data (ND) in response to receiving the second medical image (I2). The denoising model (DM) is configured to reconstruct the second medical image (I2) from the noisy data (ND). The trained conditional generative model (CGM) mentioned above t This does not include the Noise Model (NM).
10. The method according to any one of claims 1 to 9, wherein the conditional generation model (CGM) is a potential diffusion model.
11. The method according to any one of claims 1 to 10, wherein the training further comprises: - Mask at least a portion of the first medical image (I1) and / or image embedding (E1).
12. The method according to any one of claims 1 to 11, Each dataset (DS) includes at least two medical images comprising multiple first medical images (I1, I2, I3) and a second medical image (I4). At least some of the plurality of first medical images (I1, I2, I3) represent the examination areas of the object being examined at different time points. The image embedding used to generate the first medical image using the image encoder includes: - A portion of the plurality of first medical images (I1, I2, I3) is at least partially masked. - Generate image embeddings (E1, E2, E3) for each first medical image, including the portion of the first medical image that has been at least partially masked, and combine the image embeddings (E1, E2, E3) into a combined image embedding (EC). The generation of the reconstructed second medical image based on the image embedding of the first medical image and the second medical image includes: generating the reconstructed second medical image (RI4) based on the combined image embedding (EC) of the first medical image (I1, I2, I3) and the second medical image (I4), wherein the combined image embedding (EC) of the first medical image (I1, I2, I3) is used as a condition in the generation of the reconstructed second medical image (RI4).
13. The method according to any one of claims 1 to 12, Each dataset (DS) includes at least two medical images comprising a first medical image (I1, I2, I3) and a second medical image (I4) of number p, where p is 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10. At least some of the p-number of first medical images (I1, I2, I3) represent the examination areas of the object at different time points. The image embedding used to generate the first medical image using the image encoder includes: - Divide each of the p-number of first medical images (I1, I2, I3) into multiple blocks. - An image embedding (EC) is generated using the image encoder (IE) and based on selected blocks (P11, P22, P31, P43, P52, P61, P73, P83, P92) from the plurality of blocks, wherein each selected block represents a different sub-region of the inspection area. The generation of the reconstructed second medical image based on the image embedding of the first medical image and the second medical image includes: generating the reconstructed second medical image (RI4) based on the image embedding (EC) of the first medical image (I1, I2, I3) and the second medical image (I4), wherein the image embedding (EC) of the first medical image (I1, I2, I3) is used as a condition in the generation of the reconstructed second medical image (RI4).
14. A computer system (1), comprising: Processing unit (20); as well as Memory (50) storing computer program (60) configured to perform an operation when implemented by the processing unit, the operation including: - Provides an image encoder (IE), wherein the image encoder (IE) is configured to generate image embeddings based on medical images. - Provides a trained Conditional Generative Model (CGM) t The trained conditional generation model (CGM) mentioned above. t The model is trained to generate reconstructed medical images based on medical images, conditions, and model parameters, wherein the training includes: Multiple datasets are provided for multiple examination subjects, each dataset (DS) including at least two medical images: a first medical image (I1) and a second medical image (I2), wherein the first medical image (I1) represents the examination area of the examination subject at a first time point, and the second medical image (I2) represents the examination area of the examination subject at a second time point, wherein the first time point and the second time point are different time points. For each dataset (DS): The image encoder (IE) is used to generate the image embedding (E) of the first medical image (I1). Using the conditional generation model (CGM), a reconstructed second medical image (RI2) is generated based on the image embedding (E) of the first medical image (I1) and the second medical image (I2), wherein the image embedding (E) of the first medical image (I1) is used as a condition in the generation of the reconstructed second medical image (RI2). Determine the deviation between the second medical image (I2) and the reconstructed second medical image (RI2). The bias can be reduced by modifying the model parameters of the conditional generative model (CGM). - Receive at least one medical image (I) of the new patient being examined. n ), wherein at least one medical image (I n () indicates the inspection area of the new inspection object at the first time point. - Using the image encoder (IE), based on at least one medical image (I) of the new examination object. n To generate at least one image embedding (E) n ), - Using the at least one of the image embeddings (E n ) as the trained conditional generative model (CGM) t The conditions for input are used to generate synthetic medical images (SI). - Output and / or store the synthetic medical image (SI), and / or transmit the synthetic medical image (SI) to a separate computer system.
15. A non-transitory computer-readable storage medium on which a computer program (60) is stored, the computer program, when executed by a processing unit (20) of a computer system (1), causes the computer system (1) to perform the following steps: - Provides an image encoder (IE), wherein the image encoder (IE) is configured to generate image embeddings based on medical images. - Provides a trained Conditional Generative Model (CGM) t The trained conditional generation model (CGM) mentioned above. t The model is trained to generate reconstructed medical images based on medical images, conditions, and model parameters. The training includes: Multiple datasets are provided for multiple examination subjects, each dataset (DS) including at least two medical images: a first medical image (I1) and a second medical image (I2), wherein the first medical image (I1) represents the examination area of the examination subject at a first time point, and the second medical image (I2) represents the examination area of the examination subject at a second time point, wherein the first time point and the second time point are different time points. For each dataset (DS): The image encoder (IE) is used to generate the image embedding (E) of the first medical image (I1). A reconstructed second medical image (RI2) is generated based on the image embedding (E) of the first medical image (I1) and the second medical image (I2), wherein the image embedding (E) of the first medical image (I1) is used as a condition in the generation of the reconstructed second medical image (RI2). Determine the deviation between the second medical image (I2) and the reconstructed second medical image (RI2). The bias can be reduced by modifying the model parameters of the conditional generative model (CGM). - Receive at least one medical image (I) of the new patient being examined. n ), wherein at least one medical image (I n () indicates the inspection area of the new inspection object at the first time point. - Using the image encoder (IE), based on at least one medical image (I) of the new examination object. n To generate at least one image embedding (E) n ), - Using the at least one of the image embeddings (E n ) as the trained conditional generative model (CGM) t The conditions for input are used to generate synthetic medical images (SI). - Output and / or store the synthetic medical image (SI), and / or transmit the synthetic medical image (SI) to a separate computer system.
16. The use of contrast agents in dynamic examination of the examination area of an object, wherein the dynamic examination includes: - Receive one or more medical images (I n ), wherein one or more medical images (I n () indicates the examination area of the subject at one or more time points before and / or after the application of the contrast agent. - Provides an image encoder (IE), wherein the image encoder (IE) is configured to generate an image embedding based on the one or more medical images. - Using the image encoder (IE), based on the one or more medical images (I n To generate at least one image embedding (E) n ), - Provides a trained Conditional Generative Model (CGM) t ), wherein the trained conditional generative model (CGM) t It is trained to generate synthetic medical images based on image embeddings as conditions. - Using the at least one of the image embeddings (E n ) as the trained conditional generative model (CGM) t The conditions for input are used to generate synthetic medical images (SI). - Output and / or store the synthetic medical image (SI), and / or transmit the synthetic medical image (SI) to a separate computer system. The trained conditional generative model (CGM) mentioned above t The training includes: - Provide multiple datasets for multiple reference examination subjects, each dataset (DS) including at least two reference medical images: one or more first reference medical images (I1) and a second reference medical image (2), wherein the one or more first reference medical images (I1) represent the examination area of the reference examination subject at one or more time points before and / or after the application of the contrast agent, and the second reference medical image (I2) represents the examination area of the reference examination subject at another time point before and / or after the application of the contrast agent. - For each dataset (DS): use the image encoder (IE) to generate one or more reference image embeddings (E) for the one or more first reference medical images (I1). - Train the conditional generative model (CGM), wherein for each dataset (DS), the training includes: A reconstructed second reference medical image (RI2) is generated based on the one or more reference image embeddings (E) of the one or more first reference medical images (I1) and the second reference medical image (I2), wherein the one or more reference image embeddings (E) of the one or more first reference medical images (I1) are used as conditions in the generation of the reconstructed second reference medical image (RI2). Determine the deviation between the second reference medical image (I2) and the reconstructed second reference medical image (RI2). The bias is reduced by modifying the model parameters of the Conditional Generative Model (CGM).
17. The use of the contrast agent according to claim 16, wherein the contrast agent is a hepatobiliary contrast agent, and wherein the dynamic examination comprises: - Receive one or more MRI images (I1) n I2 n , I3 n ), wherein the one or more MRI images (I1 n I2 n ,I3 n () indicates the liver or a portion thereof of the subject being examined during one or more of the following periods before and / or after the application of the hepatobiliary contrast agent: plain scan phase, arterial phase, portal venous phase, transition phase, - Provide the image encoder (IE), wherein the image encoder (IE) is configured to generate an image embedding based on the one or more MRI images. - Using the image encoder (IE), based on the one or more MRI images (I1) n I2 n , I3 n To generate the at least one image embedding (E1) n E2 n E3 n ), - Provide the trained Conditional Generative Model (CGM) t ), wherein the trained conditional generative model (CGM) t It is trained to generate synthetic medical images based on image embeddings as conditions. - Using the at least one of the image embeddings (E1) n E2 n E3 n ) as the trained conditional generative model (CGM) t The conditions for input are used to generate synthetic MRI images (SI), wherein the synthetic MRI images (SI) represent the liver or a portion thereof of the examined subject during the hepatobiliary phase following the application of the hepatobiliary contrast agent. - Output and / or store the synthesized MRI images (SI), and / or transfer the synthesized MRI images (SI) to a separate computer system. The training of the conditional generative model (CGM) includes: - Provides multiple datasets for multiple reference examination objects, each dataset (DS) including at least two reference MRI images: one or more first reference MRI images (I1, I2, I3) and a second reference MRI image (I4), The one or more first reference MRI images (I1, I2, I3) represent the liver or a portion thereof of the reference subject during one or more of the following periods before and / or after the application of the hepatobiliary contrast agent: plain scan phase, arterial phase, portal venous phase, transition phase, The second reference MRI image (I4) represents the liver or a portion thereof of the reference subject during the hepatobiliary phase following the application of the hepatobiliary contrast agent. - For each dataset (DS): Use the image encoder (IE) to generate one or more reference image embeddings (E1, E2, E3) for the one or more first reference MRI images (I1, I2, I3). - Train the conditional generative model (CGM), wherein for each dataset (DS), the training includes: A reconstructed second reference MRI image (RI4) is generated based on the one or more reference image embeddings (E1, E2, E3) of the one or more first reference MRI images (I1, I2, I3) and the second reference MRI image (I4), wherein the one or more reference image embeddings (E1, E2, E3) of the one or more first reference MRI images (I1, I2, I3) are used as conditions in the generation of the reconstructed second reference MRI image (RI4). Determine the deviation between the second reference MRI image (I4) and the reconstructed second reference MRI image (RI4). The bias is reduced by modifying the model parameters of the Conditional Generative Model (CGM).
18. A kit comprising a contrast agent and a computer program, the computer program, when implemented by a processing unit (20) of a computer system (1), causing the computer system (1) to perform the following steps: - Receive one or more medical images (I1) n I2 n , I3 n ), wherein one or more medical images (I1) n I2 n , I3 n () indicates the examination area of the subject at one or more time points before and / or after the application of the contrast agent. - Provides an image encoder (IE), wherein the image encoder (IE) is configured to generate an image embedding based on the one or more medical images. - Using the image encoder (IE), generate at least one image embedding (EC) based on the one or more medical images. n ), - Provides a trained conditional generation model, wherein the trained conditional generation model (CGM) t It is trained to generate synthetic medical images based on image embeddings as conditions. - Using at least one of the image embeddings (EC) n ) as the trained conditional generative model (CGM) t The conditions for input are used to generate synthetic medical images (SI). - Output and / or store the synthetic medical image (SI), and / or transmit the synthetic medical image (SI) to a separate computer system. The trained conditional generative model (CGM) mentioned above t The training includes: - Provides multiple datasets for multiple reference examination subjects, each dataset (DS) including at least two reference medical images: one or more first reference medical images (I1, I2, I3) and a second reference medical image (I4), wherein the one or more first reference medical images (I1, I2, I3) represent the examination area of the reference examination subject at one or more time points before and / or after the application of the contrast agent, and the second reference medical image (I4) represents the examination area of the reference examination subject at another time point before and / or after the application of the contrast agent. - For each dataset (DS): Use the image encoder (IE) to generate one or more reference image embeddings (ECs) for the one or more first reference medical images (I1, I2, I3). - Train the conditional generative model (CGM), wherein for each dataset (DS), the training includes: A reconstructed second reference medical image (RI4) is generated based on the one or more reference image embeddings (ECs) of the one or more first reference medical images (I1, I2, I3) and the second reference medical image (I4), wherein the one or more reference image embeddings (ECs) of the one or more first reference medical images (I1, I2, I3) are used as conditions in the generation of the reconstructed second reference medical image (RI4). Determine the deviation between the second reference medical image (I4) and the reconstructed second reference medical image (RI4). The bias is reduced by modifying the model parameters of the Conditional Generative Model (CGM).
Citation Information
Patent Citations
Derivatized DTPA complexes, pharmaceutical agents containing these compounds, their use, and processes for their production
US6039931A
New contrast agent for use in magnetic resonance imaging
WO2022194777A1
