Medical image generation method and device, artificial intelligence model training method and device, and program

MYPI2025004731A0Undetermined Publication Date: 2024-08-07IIDA HOME MAX CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
MYPI2025004731
Authority / Receiving Office
MY · MY
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-02-07
Filing Date
2024-02-07
Publication Date
2024-08-07

AI Technical Summary

Technical Problem

Limited access to positron emission tomography (PET) scans due to resource-intensive facilities and radiation exposure, making it difficult for all patients to undergo MET-PET testing for glioma diagnosis, while MR images provide correlated information that can be leveraged for diagnostic purposes.

Method used

Developing an AI model using generative adversarial networks to generate composite PET-like images from MR images, allowing for equivalent diagnostic capabilities without the need for PET scans, by training the model with paired MR and PET images.

Benefits of technology

Enables the generation of synthetic PET images from MR images, facilitating diagnosis and prognosis prediction for patients without access to PET scans, reducing resource constraints and radiation exposure, and providing a cost-effective alternative for brain tumor differentiation and grading.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure PI2025004731_DRAWING_001
    Figure PI2025004731_DRAWING_001
  • Figure PI2025004731_DRAWING_002
    Figure PI2025004731_DRAWING_002
  • Figure PI2025004731_DRAWING_003
    Figure PI2025004731_DRAWING_003
Patent Text Reader

Abstract

ABSTRACT MEDICAL IMAGE GENERATION METHOD AND DEVICE, ARTIFICIAL INTELLIGENCE MODEL TRAINING METHOD AND DEVICE, AND PROGRAM An object of the present invention is to generate a synthetic image which is similar to a positron emission tomography (PET) image from a magnetic resonance (MR) image of a patient. One embodiment of the present invention provides, a medical image generation method characterized by causing execution of the steps of: acquiring a magnetic resonance (MR) image of a particular patient suspected of having a certain disease; and generating, from the MR image of the particular patient, a synthetic image similar to a positron emission tomography (PET) image, by using a learned model obtained by learning an artificial intelligence model by using, as learning data, MR images of patients diagnosed to have the disease and the PET images corresponding to the MR images. FIG. 6
Need to check novelty before this filing date? Find Prior Art

Description

Medical image generation method and device, artificial intelligence model learning method and device, and program

[0001] The present invention relates to a method, apparatus, and program for generating a composite image similar to positron emission tomography (PET) from magnetic resonance (MR) images of a patient.

[0002] 11 C-Methionine positron emission tomography (MET-PET) is performed on patients with suspected brain tumors to assess methionine accumulation in areas of active amino acid metabolism. MET-PET is useful for glioma grading, prognosis prediction, and tumor extent delineation. Recent guidelines support the use of MET-PET for the preoperative evaluation of patients with glioma. However, MET-PET requires a cyclotron and scanner, and therefore, access to the test is limited, and not all patients suspected of having glioma can undergo MET-PET. Furthermore, MET-PET exposes patients to radiation.

[0003] It has been reported that both methionine accumulation and contrast-enhanced T1-weighted enhancement on CE-MR imaging of gliomas indicate cell proliferation and angiogenesis (Non-Patent Documents 1-4). This suggests that the features of MET-PET and CE-MR are correlated with each other.

[0004] In recent years, deep learning image-to-image translation techniques have been developed (Non-Patent Documents 5-8). Image-to-image translation models involve the application of generative adversarial networks that can extract features between images (Non-Patent Document 9). For example, computed tomography images for attenuation correction are [ 18 F]-2-fluoro-2-deoxy-D-glucose positron emission tomography (FDG-PET) images (Non-Patent Document 10), and brain MR images have been generated from brain FDG-PET images (Non-Patent Document 11).

[0005] ​Langen KJ, Muhlensiepen H, Holschbach M, et al: Transport mechanisms of 3-[123I]iodo-alpha-methyl-L-tyrosine in a human glioma cell line: comparison with [3H]methyl]-L-methionine. J Nucl Med 41:1250-5, 2000Kracht LW, Friese M, Herholz K, et al: Methyl-[11C]- l-methionine uptake as measured by positron emission tomography correlates to microvessel density in patients with glioma. Eur J Nucl Med Mol Imaging 30:868-73, 2003Tynninen O, Aronen HJ, Ruhala M, et al: MRI enhancement and microvascular density in gliomas. Correlation with tumor cell proliferation. Invest Radiol 34:427-34, 1999Sadeghi N, Salmon I, Decaestecker C, et al: Stereotactic comparison among cerebral blood volume, methionine uptake, and histopathology in brain glioma. AJNR Am J Neuroradiol 28:455-61, 2007LeCun Y, Bengio Y, Hinton G: Deep learning. Nature 521:436-44, 2015Hinton G: Deep Learning-A Technology With the Potential to Transform Health Care.JAMA 320:1101-1102, 2018Mirza M, Osindero S: Conditional Generative Adversarial Nets, 2014, pp arXiv:1411.1784Ueda D, Shimazaki A, Miki Y: Technical and clinical overview of deep learning in radiology. Jpn J Radiol 37:15-33, 2019Mirza M, Osindero S: Conditional Generative Adversarial Nets. arXiv e-prints:arXiv:1411.1784, 2014Armanious K, Jiang C, Fischer M, et al: MedGAN: Medical image translation using GANs. Comput Med Imaging Graph 79:101684, 2020Bazangani F, Richard FJP, Ghattas B, et al: FDG-PET to T1 Weighted MRI Translation with 3D Elicit Generative Adversarial Network (E-GAN). Sensors (Basel) 22, 2022.

[0006] In view of the potential correlation between magnetic resonance (MR) and positron emission tomography (PET) images, the inventors have considered the possibility of generating PET-like images from MR images of patients by using image-to-image conversion technology, and have arrived at the present invention. Here, the target disease is not limited to glioma, but may be other tumors or even other diseases. Furthermore, the patient is not limited to humans, but may also be an animal. Furthermore, as long as it can be correlated with MR images, 11 The present invention is not limited to C-Methionine PET images, but may also be applicable to other types of PET images. That is, an object of the present invention is to provide a method, an apparatus, and a program for generating a PET-like composite image from a patient's MR images, including training an artificial intelligence model for generating the composite image.

[0007] In order to solve the above-mentioned problems, a first aspect of the present invention provides a medical image generation method, characterized by executing the steps of: acquiring magnetic resonance (MR) images of a specific patient suspected of having a certain disease; and using a trained model obtained by training an artificial intelligence model using MR images of patients diagnosed with the disease and positron emission tomography (PET) images corresponding to the MR images as training data, generating a synthetic image similar to a PET image from the MR images of the specific patient using the trained model.

[0008] In the medical image generation method of the present invention, it is preferable that the artificial intelligence model includes a generator that generates a composite image from an input MR image, and a classifier that accepts a pair of the input MR image and the generated composite image, or a pair of the input MR image and a PET image corresponding to the input MR image, and determines whether the pair is true or false.

[0009] In addition, in the medical image generating method of the present invention, it is preferable that the learning data is composed of a set of multiple slices in an MR image and one slice in a PET image corresponding to the MR image.

[0010] In the medical image generating method of the present invention, it is preferable that the learning data be configured as a pair of an entire MR image and a cut-out image in which a lesion area is cut out from the entire MR image.

[0011] In addition, a second aspect of the present invention provides a program for executing the steps of acquiring magnetic resonance (MR) images of a specific patient suspected of having a certain disease, and generating a synthetic image similar to a PET image from the MR images of the specific patient using a trained model obtained by training an artificial intelligence model using MR images of patients diagnosed with the disease and positron emission tomography (PET) images corresponding to the MR images as training data.

[0012] A third aspect of the present invention provides a medical image generation device comprising: an acquisition unit that acquires magnetic resonance (MR) images of a specific patient suspected of having a certain disease; a memory unit that stores a trained model obtained by training an artificial intelligence model using MR images of a patient diagnosed with the disease and positron emission tomography (PET) images corresponding to the MR images as training data; and an image generation unit that uses the trained model to generate a synthetic image similar to a PET image from the MR images of the specific patient.

[0013] A fourth aspect of the present invention provides an artificial intelligence model training method comprising the steps of: acquiring magnetic resonance (MR) images of a group of patients diagnosed with a specific disease and positron emission tomography (PET) images corresponding to the MR images; training an artificial intelligence model using the acquired MR images and PET images as training data; and generating a trained model that generates a synthetic image similar to the PET image from the MR images of patients suspected of having the disease.

[0014] A fifth aspect of the present invention provides a program for executing the steps of acquiring magnetic resonance (MR) images of a group of patients diagnosed with a specific disease and positron emission tomography (PET) images corresponding to the MR images, and training an artificial intelligence model using the acquired MR images and PET images as training data to generate a trained model that generates a synthetic image similar to the PET image from the MR images of patients suspected of having the disease.

[0015] A sixth aspect of the present invention provides an artificial intelligence model training device comprising: a storage unit that stores magnetic resonance (MR) images of a group of patients diagnosed with a specific disease and positron emission tomography (PET) images corresponding to the MR images; and a training unit that trains an artificial intelligence model using the stored MR images and PET images as training data to generate a trained model that generates a synthetic image similar to the PET image from the MR images of patients suspected of having the disease. Here, each of the above methods is a computer-implemented method. Furthermore, each of the above programs may be stored in a non-transitory computer-readable storage medium and executed by a processor.

[0016] According to the present invention, a composite image similar to a PET image can be generated from a patient's MR images. This allows for a diagnosis equivalent to that of a PET examination to be provided to patients who have only undergone an MR examination. Therefore, more comprehensive diagnostic information can be provided to patients with limited resources.

[0017] 1 is a block diagram showing the software configuration of the medical image generation device 10. FIG. 2 is a flowchart showing a procedure for generating a medical image. FIG. 3 is a block diagram showing the functional configuration of the artificial intelligence model learning device 20. FIG. 4 is a flowchart showing a procedure for learning an artificial intelligence model. FIG. 5 is a block diagram showing the hardware configuration of a computer constituting the medical image generation device 10 and the artificial intelligence model learning device 20. FIG. 6 is a diagram showing an overview of a learning model. FIG. 7 is a diagram showing an example of registration of a positron emission tomography (PET)-magnetic resonance (MR) image pair. FIG. 8 is a diagram showing an example of displaying the relative value of each pixel. FIG. 9 is a diagram showing an example of a pair of three MR slices and one PET slice. FIG. 10 is a diagram showing an example of creating a cropped lesion image. FIG. 11 is a diagram showing examples of a CE-MR image, a synthetic MET-PET image, and an actual MET-PET image. FIG. 12 is a graph showing the results of a receiver operating characteristic (ROC) analysis and an overall survival (OS) analysis. FIG. 13 is an explanatory diagram of U-Net.

[0018] Representative embodiments of the present invention will be described in detail below with reference to the drawings. However, the present invention is not limited to these drawings. Furthermore, since the drawings are intended to conceptually explain the present invention, dimensions, ratios, or numbers may be exaggerated or simplified as necessary to facilitate understanding.

[0019] 1. Medical Image Generation Device 10 and Method The medical image generation device 10 is a computer that generates a positron emission tomography (PET) image-like composite image (hereinafter sometimes referred to as a composite PET image) from a patient's magnetic resonance (MR) image. The medical image generation device 10 may be a single computer or may be configured as a system including multiple computers. In this embodiment, the patient is assumed to be a human, but the patient may also be an animal. Furthermore, the expected disease is tumors, particularly brain tumors, but is not limited thereto, and other diseases such as epilepsy and Alzheimer's disease can also be used. While a contrast-enhanced T1-weighted (CE-MR) image is assumed as the MR image, other images such as plain T1-weighted images, T2-weighted images, and diffusion-weighted images can also be used.

[0020] As shown in FIG. 1 , the medical image generating device 10 includes an acquisition unit 11, an image generation unit 13, a trained model storage unit 15, and an image output unit 17. The acquisition unit 11 acquires MR images of a specific patient. The acquisition unit 11 may be an MR imaging device that generates MR images, or a memory that receives and stores MR images from an MR imaging device or an external device via a communication interface. The image generation unit 13 generates a composite PET image from the MR images of the specific patient using a trained model stored in the trained model storage unit 15. Here, the trained model is obtained by training an artificial intelligence model using MR images of a patient group and PET images corresponding to the MR images as training data. Details of the artificial intelligence model and training will be described later. The image output unit 17 transmits the generated composite PET image to an external device, prints it, or displays it on a display.

[0021] Referring to FIG. 2 , a general procedure for generating a medical image will be described. This procedure can be executed by a computer functioning as the medical image generation device 10. In step S11, MR images of a specific patient suspected of having a certain disease are acquired. Step S11 may be executed by the acquisition unit 11 described above. In step S12, a synthetic PET image is generated from the acquired MR images using the trained model. Step S12 may be executed by the image generation unit 13 described above. In step S13, the generated synthetic PET image is output. Step S13 may be executed by the image output unit 17 described above. The synthetic PET image can be used by a doctor to examine and diagnose the patient. The synthetic PET image is particularly useful when a PET device is not available.

[0022] 2. Artificial Intelligence Model Learning Device 20 and Method The artificial intelligence model learning device 20 is a computer that generates a trained model, and may be a single computer or may be configured as a system including multiple computers. The artificial intelligence model learning device 20 may be a single computer integrated with the medical image generation device 10, or may be a computer separate from the medical image generation device 10. As shown in FIG. 3 , the artificial intelligence model learning device 20 includes a learning unit 21, a training data storage unit 23, an artificial intelligence model storage unit 25, and a model output unit 27.

[0023] The learning unit 21 trains an artificial intelligence model using the training data to generate a trained model. In this embodiment, a generative adversarial network (GAN), a type of generative model among deep learning technologies, is preferably used. However, other generative models, such as a variational autoencoder (VAE) or a diffusion model, may be used as long as they can generate PET-like images from MR images. Here, the GAN includes a generator and a classifier. The generator generates a synthetic image similar to a PET image from an MR image. The classifier receives the synthetic image generated by the generator and a real PET image and determines whether they are true or false (see FIG. 6 ). By alternately competing these two networks and progressing through learning, the generator can generate synthetic images that resemble real PET images. More specifically, the competitive relationship between the two networks is expressed by sharing a loss function L, expressed by the following equation: That is, for a pair of images (x, y), the generator G (which generates image G(x) from image x) is trained to reduce the value of the loss function L, and the classifier D is trained to increase the value of the loss function L. In this embodiment, among GANs, pix2pix, which is a model that converts features between images, is preferably used, but is not limited to this. When the pix2pix model is used, learning of G is performed by minimizing the following equation, which takes the L1 distance into account in the loss function L(G, D), so that image G(x) is as close as possible to image x.

[0024] The training data storage unit 23 stores training data. The training data is a set of MR images and PET images of a patient diagnosed with a certain disease, and preferably has undergone preprocessing. Major preprocessing steps may include matching multiple slices of the MR images with one slice of the PET image, and generating a cutout image by cutting out or trimming only the lesion area from full-size MR and PET images (full images). The MR image corresponding to one slice of the PET image preferably consists of two or more consecutive slices, and more preferably three or more consecutive slices. Details of the preprocessing will be described later. Note that a "slice" refers to a cross-section of any region in the three-dimensional data obtained by MR or PET imaging. The artificial intelligence model storage unit 25 stores the artificial intelligence model and the trained model. The model output unit 27 stores the generated trained model in an internal or external storage medium or transmits it to an external computer such as the medical image generation device 10.

[0025] Referring to FIG. 4 , a general procedure for training an artificial intelligence model will be described. This procedure can be executed by a computer functioning as the artificial intelligence model training device 20. In step S21, MR images and PET images of a patient are collected and registered as training data. Step S21 can be implemented by the training data storage unit 23. Necessary preprocessing may be performed before the images are registered. The main preprocessing is as described above, and other preprocessing may include image resizing, reslicing, and normalization. In step S22, the artificial intelligence model is trained using the registered training data to obtain a trained model. Step S22 can be implemented by the training unit 21. In step S23, the obtained trained model may be stored and transmitted to another device, such as the medical image generation device 10. Step S23 can be implemented by the artificial intelligence model storage unit 25 and the model output unit 27.

[0026] 3. Computer Constituting the Medical Image Generation Device 10 and the Artificial Intelligence Model Learning Device 20 Fig. 5 shows an example of the hardware configuration of a computer 100 constituting the medical image generation device 10 and the artificial intelligence model learning device 20. As shown in the figure, the computer 100 includes a processor 101, a memory 103, and a communication interface 105, and may further include an input device 107 and an output device 109.

[0027] The processor 101 reads various programs and data into the memory 103 and executes them to realize various functions of the medical image generating device 10 and the artificial intelligence model learning device 20. The processor 101 may be composed of a semiconductor integrated circuit such as a central processing unit (CPU), a graphics processing unit (GPU), or a microprocessor. The memory 103 is a random access memory (RAM) and a read-only memory (ROM) that store various data and programs. The memory 103 is a non-transitory computer-readable storage medium, such as a hard disk drive, a solid-state drive, or a flash memory. The communication interface 105 is an interface for connecting to wired and wireless communication networks, such as an adapter for connecting to an Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, or a USB (Universal Serial Bus) connector or an RS232C connector for serial communication. The input device 107 is, for example, a keyboard, a mouse, a touch panel, buttons, a microphone, or the like, for inputting various data. The output device 109 outputs various types of data and is, for example, a display, a printer, a speaker, etc. The output device 109 may transmit various types of data to an external computer or external device via the communication interface 105.

[0028] 4. Advantages of the Present Embodiment According to the present embodiment, a composite image similar to a PET image can be generated from a patient's MR images. PET examinations are resource-intensive, requiring a cyclotron, a PET scanner, medical staff, and time, which can lead to high medical costs. Furthermore, PET examinations involve patient exposure to radiation. In contrast, composite PET images can be obtained from more commonly available MR images without the above-mentioned inconveniences or additional examinations. Therefore, composite PET images are expected to facilitate the differentiation and prognosis of diseases such as brain tumors when a PET examination is not available.

[0029] 5. Examples Examples of this embodiment will be described with reference to FIGS. 6 to 13. Here, we will exemplify a situation in which machine learning is performed to generate a PET-like composite image from contrast-enhanced T1-weighted (CE-MR) images obtained by an MR examination performed on a patient suspected of having glioma. However, this method is not limited to the diagnosis of glioma, and can also be applied to the diagnosis of tumors other than glioma, and even other diseases. In addition, as training data, contrast-enhanced T1-weighted images and 11 C-Methionine PET (MET-PET) imaging is used, 18 Other PET images may also be used, such as F-fluorodeoxyglucose (FDG) PET images.

[0030] 5-1. Overview of the Example The inventors collected data from patients who underwent both MET-PET and CE-MR examinations. MET-PET and CE-MR images from the same patients were paired. These image pairs were used to train, validate, and test an image-to-image conversion AI model. The trained model was then externally validated, demonstrating its performance in grading and prognostic prediction of synthetic MET-PET images generated from CE-MR images of glioma patients. The following describes the example in detail.

[0031] 5-2. Deep Learning and Datasets Deep learning has three main phases: training, validation, and testing. It is common to prepare training and validation datasets for training and adjusting an AI model. The test dataset is used to test the model. To avoid overestimating and overfitting the model, the same data and images are not used in any two of the above datasets.

[0032] Deep learning training is typically performed on randomly sampled batches (i.e., subsets) of the training dataset, and these batches are used to optimize the model weights via backpropagation. Training is performed by iteratively updating the model parameters until the model best fits the data. A validation dataset is used for parameter selection and tuning, and can also be used to enforce stopping criteria for training.

[0033] In the testing phase, an independent dataset is used to evaluate the discrimination performance of the AI ​​model, which ensures the model's generalizability. An internal testing dataset is an independent dataset used to evaluate the model's performance, which is not used in the training phase but is obtained from the same facility as the data used in the training phase. An external testing dataset is a dataset for performance evaluation that is not used in the training phase and is collected from another facility.

[0034] 5-3. Dataset Preparation First, we prepared an in-house dataset (herein referred to as the internal dataset) for training, validating, and internally testing the AI ​​model. Here, "this facility" refers to Osaka Municipal University Hospital. Specifically, patients who underwent both MET-PET and CE-MR examinations at this facility between January 2007 and December 2018 were retrospectively collected. However, patients who met certain exclusion criteria were excluded. Pathological diagnoses were determined pathologically if the patient underwent surgery; otherwise, they were determined according to international guidelines for each disease.

[0035] The details of the CE-MR examination are as follows: Patients received 0.1 ml / kg of gadobutrol (Gadovist®) or 0.2 ml / kg of meglumine gadoterate (Magnescope®) or gadoteridol (ProHance®) intravenously. CE-MR images were scanned as routine examinations. The acquisition parameters were as follows: echo time 1.7-20 msec; repetition time 4.0-2020 msec; matrix 256x256-640x640; and slice thickness 0.8-5 mm. The details of the MET-PET examination are as follows: Patients were fasting and: 11 C-Methionine was administered intravenously at 6 MBq / kg over 30 seconds. After the transmission scan, a 10-second static scan was initiated 20 minutes after administration. The acquisition parameters were as follows: matrix 128x128-336x336; slice thickness 2.0-3.2 mm.

[0036] Next, data (referred to here as external datasets) were prepared for external validation of the trained model. The external datasets consisted of MR images, pathological diagnosis, grading, and prognosis data from patients diagnosed with primary glioma who underwent preoperative MR examinations but not MET-PET examinations. In this example, the following open-source datasets were used: The Cancer Genome Atlas Low Grade Glioma Collection (http: / / doi.org / 10.7937 / K9 / TCIA.2016.L4LTD3TK); and The Cancer Genome Atlas Glioblastoma Multiforme Collection (https: / / doi.org / 10.7937 / K9 / TCIA.2016.RNYFUYE9). These external datasets were collected from multiple institutions in the United States and Italy. However, patients meeting certain exclusion criteria were excluded.

[0037] The details of the MRI examination were as follows. The type of contrast agent was not disclosed and was unknown. CE-MR images were scanned with the following acquisition parameters: echo time 1.6-20 msec; repetition time 4.7-3285 msec; matrix 240x240-512x512; and slice thickness 0.9-6 mm.

[0038] 5-4. Data Distribution All patient data in the internal dataset was randomly allocated to the training, validation, and internal testing datasets in an 8:1:1 ratio. This patient-based distribution ensured that there was no overlap of images or patients between each dataset. All patients in the external dataset were from a different country than our institution and were used to demonstrate the generalization of our AI model.

[0039] 5-5. Ground Truth Labeling. In this example, lesions were defined as voxels in MET-PET images with pixel values ​​1.3 times or greater than the mean pixel value of the contralateral cerebral cortex with normal appearance. Two radiologists (blind to the patient's diagnosis and prognosis) segmented the PET-PET and CE-MR images as follows, and another radiologist (blind to the patient's diagnosis and prognosis) adjusted the images to create the final segmentation. All regions of interest (ROIs) were generated using ITK-SNAP. a) Reference ROIs in the Cerebral Hemisphere for MET-PET Images. To normalize the MET-PET images, the signal intensity of the normal-appearing cerebral cortex contralateral to the lesion was used as the reference. More specifically, five spherical ROIs with a diameter of 1 cm were placed in the contralateral cerebral hemisphere that appeared normal on the MET-PET images. Alternatively, if the lesion was located in both cerebral hemispheres, five spherical ROIs were placed in both normal-appearing cerebral cortices. b) Reference ROIs in the Cerebellar Hemisphere for CE-MR Images. To normalize the CE-MR images, the signal intensity of normal-appearing cerebellar white matter was used as a reference. More specifically, two spherical ROIs with a diameter of 1 cm were placed in the left and right cerebellar white matter on the CE-MR images. Alternatively, if there was a lesion in either the left or right cerebellar white matter, two ROIs were placed in the contralateral cerebellar white matter.

[0040] This process allows MET-PET and CE-MRI to be expressed in relative pixel values, thereby absorbing differences in pixel representation due to differences in image capture equipment and software processing. Figure 8 shows examples of images before and after normalization. Panels (A) and (B) show the MR and PET images, respectively. The upper panel shows the original image, providing the basis for correcting for differences between images. The maximum and mean pixel values ​​of the lesion on the MET-PET image divided by the mean pixel value of the contralateral cerebral cortex with a normal appearance were defined as the normalized uptake value ratio (SUVR) maximum and mean SUVR, respectively.

[0041] 5-6. Creation of Learning Model 5-6-1. Preprocessing In this example, the AI ​​model was developed based on the pix2pix model, a generative adversarial network (GAN) that performs image-to-image translation using features extracted between image pairs (see Figure 6). However, this model differs from previous pix2pix developments in two key areas: (a) the method for handling three-dimensional (3D) data, and (b) the data sampling method. These differences enable efficient feature learning of methionine accumulation in lesions.

[0042] Regarding (a) above, the present model is designed to generate one synthetic MET-PET image using three consecutive slices of a CE-MR image. Panels (A) and (B) of Figure 9 show examples of MR and PET images, respectively. Three consecutive MR image slices are paired with one PET image slice. This procedure is a technique for efficiently learning 3D data. In other words, typical image-to-image conversion models are primarily intended for converting between two-dimensional images, such as generating a map from an aerial photograph, and are not designed to handle three-dimensional image information. This example deals with three-dimensional information, such as the continuity of lesions. From this perspective, the inventors paired consecutive slices (n-1, n, n+1) of a CE-MR image with a single slice 1(n) of a MET-PET image when creating the paired images (where n is a natural number indicating the slice number of the image data).

[0043] Regarding (b) above, two basic image pairs are first prepared: a cropped image pair and a full image pair. The cropped image pair consists of three consecutive CE-MR image slices and one MET-PET image slice of the lesion only (the remaining area outside the cropped area may be padded with black). The full image pair consists of three consecutive CE-MR image slices and one MET-PET image slice of the full image (i.e., an image without cropping). Figure 10 shows an example of a cropped image pair and a full image pair. Using both full and cropped images in this training process enabled the extraction of features of both normal and abnormal methionine accumulation. The data sampling methods (a) and (b) above enable the AI ​​model to efficiently learn both local lesion information and information from the entire brain.

[0044] Therefore, the registered CE-MR image and MET-PET image pair serve as the base pair for training data. The registered image pair includes adjustments for alignment, slice range, imaging angle, and patient orientation. Figure 7 shows an example of images before and after registration. Panel (A) shows the original image pair before registration. Panel (A-1) shows the reference PET image, and Panel (A-2) shows the original MR image. Panel (B) shows an example of a transformed image pair after registration. Panel (B-1) shows the same image as Panel (A-1), but Panel (B-2) shows the registered MR image resliced ​​based on the reference PET image (A-1). The white line in the figure indicates the position of the axial plane.

[0045] After these preprocessing steps, we trained the pix2pix model. The model was trained, tuned, and evaluated on the training, validation, and internal test datasets, respectively. The entire network was used for pixel-by-pixel regression and end-to-end mapping.

[0046] FIG. 6 is an explanatory diagram of training of a deep learning model. The deep learning model in this embodiment was developed based on pix2pix, a type of generative adversarial network. This is an image-to-image translation model that uses paired images in training and validation datasets. The deep learning model employs a U-Net-based architecture as the generator and a convolutional PatchGAN classifier as the discriminator.

[0047] Here, U-Net is a full-layer convolutional network for image segmentation, consisting of an encoder and a decoder (see Figure 13). The encoder convolves the input image multiple times to extract its features. The encoder can utilize the structure of an image classification model such as ResNet. The decoder, in turn, receives the features extracted by the encoder, performs deconvolution (upsampling), and outputs a probability map of the same size as the input image. Additionally, U-Net connects the encoder's feature map to the decoder's feature map at each layer. This allows information from the large feature map on the encoder side to be transmitted to the decoder, making it easier to capture object position information during upsampling. The PatchGAN classifier decomposes the input image into patches with NxN resolution and classifies each patch as real or fake. The classifier's output is then the average truth value of all patches.

[0048] Model training consists of two phases: a sampling phase and a model development phase. In the sampling phase, a data loader sampler inputs data from full image pairs and cropped image pairs into the model based on a preset ratio. That is, the sampler adjusts the ratio of cropped image pairs to full image pairs, contributing to efficient learning of methionine-rich regions. More specifically, the images used here are obtained from patients undergoing MET-PET examinations, and most images contain lesions. However, lesions are present in only a few slices, and many more slices are lesion-free. Therefore, if the model is trained using all images, the image-to-image translation model may generate more normalized synthetic MET-PET images, taking into account the normal-rich class imbalance. Conversely, if the model is trained only using images with lesions, the model may generate synthetic MET-PET images without learning normal methionine uptake due to the disease-rich class imbalance. To correct these class imbalances, the inventors devised a sampler to feed images into the image-to-image translation model, enabling more efficient extraction of features from both lesions and normal tissue. In this example, when feeding data to the image-to-image transformation model during training, the ratio of cropped image pairs to full image pairs was adjusted from 1:9 to 9:1 (9 ways).

[0049] The model development phase then consists of three steps: Step 1, the image generation phase; Step 2, the classifier training phase; and Step 3, the generator training phase. In Step 1, the generator generates one synthetic MET-PET image from three consecutive CE-MR images. The CE-MR image is then concatenated with the synthetic MET-PET image (see the dash-dotted line in Figure 6). In Step 2, the concatenated synthetic MET-PET and CE-MR images from Step 1 or the concatenated real MET-PET and CE-MR images from the training dataset are input to the classifier (see the solid line in Figure 6). The classifier is used to correctly distinguish between the synthetic and real MET-PET images. Therefore, if the classifier is correct, a small loss value is set; if the classifier is incorrect, a large loss value is set. The resulting classifier loss value is backpropagated to the classifier to update its parameters. In step 3, the generator is used to generate synthetic MET-PET images that are so similar to the actual MET-PET images that the classifier misidentifies them (see the dashed line in Figure 6). Therefore, if the classifier is correct, the loss value is set to be large, and if the classifier is incorrect, the loss value is set to be small. Furthermore, the L1 loss values ​​from the actual MET-PET image and the synthetic MET-PET image are obtained. These two loss values ​​are combined to generate the loss value for the generator, which then updates the generator parameters. This process is repeated to progress the learning and obtain a trained model.

[0050] The trained model takes a patient's MR image as input and outputs a synthetic image that resembles the MR image. Figure 11 shows an example of a CE-MR image, a synthetic MET-PET image, and a real MET-PET image from a patient with glioma and diffuse astrocytoma.

[0051] 5-7. Statistical Analysis of Generated Images. Pearson's correlation coefficient was calculated for the correlation between SUVR max, SUVR mean, and lesion volume between synthetic and real MET-PET images. Receiver operating characteristic (ROC) analysis and overall survival (OS) analysis were performed using the external testing dataset. Figure 12 shows the results. In the ROC analysis, the area under the curve (AUC) was calculated using SUVR max and SUVR mean on the synthetic MET-PET images to estimate the accuracy of distinguishing between low-grade and high-grade gliomas. OS analysis was performed using the Kaplan-Meier method, and patients were stratified into high-risk and low-risk groups based on the mean SUVR max and SUVR mean. Stratification performance was evaluated using the log-rank test. OS was defined as the time between the date of glioma diagnosis and the date of death. If death could not be confirmed, the patient was defined as censored. The follow-up period for censored patients was the time between the date of glioma diagnosis and the date of last contact. Additionally, we analyzed the maximum and mean SUVR in high-grade and low-grade gliomas for both internal and external testing datasets. Statistical significance was set at P<0.05.

[0052] 5-7-1. Demographic Characteristics The internal dataset consisted of 32,075 image pairs from 475 MET-PET and CE-MR examinations in 362 patients. The training dataset included 25,848 image pairs from 384 examinations in 294 patients (mean age ± standard deviation [SD] 48 years ± 18 years; 44% male). The validation dataset included 3,099 image pairs from 46 examinations in 34 patients (mean age ± SD 54 years ± 17 years; 52% male). The internal testing dataset included 3,128 image pairs from 45 examinations in 34 patients (mean age ± standard deviation [SD] 50 years ± 17 years; 50% male). The external dataset included 41,755 images from 344 CE-MR examinations in 344 patients (mean age ± standard deviation [SD] 53 years ± 15 years; 55% male) from two public datasets from The Cancer Imaging Archive.

[0053] 5-7-2. Model Development Results The Pearson correlation coefficients between synthetic and real MET-PET images were 0.68 (95% confidence interval [CI]: 0.47-0.81) for maximum SUVR, 0.76 (95% CI: 0.59-0.86) for mean SUVR, and 0.92 (95% CI: 0.85-0.95) for lesion volume, with P values ​​less than 0.001 for all. Generally, Pearson correlation coefficient values ​​range from -1 to 1, with absolute values ​​of 0-0.19 indicating very weak correlation, 0.2-0.39 indicating weak correlation, 0.40-0.59 indicating moderate correlation, 0.6-0.79 indicating strong correlation, and 0.8-1 indicating very strong correlation, respectively.

[0054] 5-7-3. Model evaluation in external population results. The ROC for the discrimination between low-grade and high-grade gliomas was 0.81 (95% CI: 0.75-0.86) for maximum SUVR and 0.78 (95% CI: 0.73-0.84) for mean SUVR. Furthermore, the group with a mean SUVR higher than 1.76 had a significantly shorter OS than the group with a lower value (P<0.005).

[0055] 5-7-4. Evaluation The inventors developed an artificial intelligence model for generating synthetic PET images from CE-MR images. In an internal testing dataset, Pearson's correlation coefficients showed strong to very strong correlations between synthetic and real MET-PET images for SUVR max, SUVR mean, and lesion volume. Regarding differentiation between low-grade and high-grade gliomas, ROC analysis of SUVR max and SUVR mean showed a high AUC. The group with SUVR max and SUVR mean higher than the mean had a significantly shorter OS than the group with values ​​lower than the mean. These results demonstrate the clinical usefulness of synthetic PET images generated from CE-MR images.

[0056] To the inventors' knowledge, this is the first example of generating a synthetic PET image from an MR image. MRI examinations are usually used for static evaluations, as they are suitable for evaluating the morphology and properties of organs. In contrast, PET examinations can be used for dynamic evaluations of the metabolic state of lesions. In this respect, MR and PET examinations are complementary. The inventors have successfully generated a dynamic PET image from a static MR image. This suggests that MR images may contain metabolic information of lesions.

[0057] PET examinations are resource-intensive in terms of requiring a cyclotron, a PET scanner, medical staff, and time, leading to high medical costs. Furthermore, the examination involves radiation exposure to the patient. In contrast, composite PET images can be obtained from MR images without the above-mentioned inconveniences or additional examinations, since MR images are generally obtained in routine clinical situations for patients with suspected brain tumors. When PET examinations are not possible, this model will facilitate the differentiation and prognosis of brain tumors.

[0058] In conclusion, this AI model enables the generation of synthetic PET images from MR images. The synthetic PET images are highly correlated with actual PET images and show significant performance for glioma grading and prognosis. This AI model can generate clinically useful images without radiation exposure from MR images, making it particularly useful for patients with gliomas in resource-limited settings.

[0059] Although typical embodiments of the present invention have been described above, the present invention is not limited to these, and various design modifications are possible, which are also included in the present invention.

[0060] REFERENCE SIGNS LIST 10 Medical image generation device 11 Acquisition unit 13 Image generation unit 15 Trained model storage unit 17 Image output unit 20 Artificial intelligence model learning device 21 Learning unit 23 Training data storage unit 25 Artificial intelligence model storage unit 27 Model output unit

Claims

1. A medical image generation method comprising the steps of: acquiring magnetic resonance (MR) images of a specific patient suspected of having a certain disease; and using MR images of patients diagnosed with the disease and positron emission tomography (PET) images corresponding to the MR images as training data, training an artificial intelligence model to obtain a trained model, thereby generating a synthetic image similar to a PET image from the MR images of the specific patient.

2. The medical image generating method according to claim 1, characterized in that the artificial intelligence model includes: a generator that generates a synthetic image from an input MR image; and a classifier that accepts a pair of an input MR image and a generated synthetic image, or a pair of an input MR image and a PET image corresponding to the input MR image, and determines whether the image is true or false.

3. The medical image generating method according to claim 2, wherein the learning data is composed of a set of a plurality of slices in an MR image and one slice in a PET image corresponding to the MR image.

4. The medical image generating method according to claim 2, wherein the learning data is configured as a pair of an entire MR image and a cut-out image in which a lesion area is cut out from the entire MR image.

5. A program for executing the steps of acquiring magnetic resonance (MR) images of a specific patient suspected of having a certain disease, and using MR images of patients diagnosed with the disease and positron emission tomography (PET) images corresponding to the MR images as training data, training an artificial intelligence model to obtain a trained model, thereby generating a synthetic image similar to a PET image from the MR images of the specific patient.

6. A medical image generating device comprising: an acquisition unit that acquires magnetic resonance (MR) images of a specific patient suspected of having a certain disease; a memory unit that stores a trained model obtained by training an artificial intelligence model using MR images of patients diagnosed with the disease and positron emission tomography (PET) images corresponding to the MR images as training data; and an image generating unit that uses the trained model to generate a synthetic image similar to a PET image from the MR images of the specific patient.

7. An artificial intelligence model training method comprising the steps of: acquiring magnetic resonance (MR) images of a group of patients diagnosed with a specific disease and positron emission tomography (PET) images corresponding to the MR images; training an artificial intelligence model using the acquired MR images and PET images as training data; and generating a trained model that generates synthetic images similar to PET images from MR images of patients suspected of having the disease.

8. A program for executing the steps of acquiring magnetic resonance (MR) images of a group of patients diagnosed with a specific disease and positron emission tomography (PET) images corresponding to the MR images, training an artificial intelligence model using the acquired MR images and PET images as training data, and generating a trained model that generates synthetic images similar to PET images from MR images of patients suspected of having the disease.

9. An artificial intelligence model learning device comprising: a memory unit that stores magnetic resonance (MR) images of a group of patients diagnosed with a specific disease and positron emission tomography (PET) images corresponding to the MR images; and a learning unit that trains an artificial intelligence model using the stored MR images and PET images as learning data to generate a trained model that generates synthetic images similar to PET images from MR images of patients suspected of having the disease.