Contrast-agent-free simplified liver magnetic resonance imaging method based on generative AI

Abstract

The invention aims to provide an algorithm based on a generative artificial intelligence model, the algorithm takes an enhanced pre-liver MRI image as input, does not need a contrast agent, efficiently and quickly outputs multi-phase enhanced images (an arterial phase, a portal vein phase, a delay phase and a liver and gall specific phase) of the liver, and forms a simplified liver scanning scheme in combination with T2WI and DWI. In order to achieve the purpose, the technical scheme of the invention provides a simplified liver magnetic resonance imaging method which is based on generative AI and does not need a contrast agent. According to the method, a deep learning model based on a stable diffusion model (SD) is established, a pre-enhancement MRI image is taken as input, each phase enhancement image is taken as a gold standard training model, and after model training is completed, any pre-enhancement image can be input, so that a four-phase liver enhancement image can be output by the model. And a simplified liver magnetic resonance imaging scheme can be formed by combining conventional T2WI and DWI acquired by MRI scanning.

Description

Technical Field

[0001] The present invention relates to a simplified liver magnetic resonance imaging scheme that does not require contrast agents and is based on generative artificial intelligence using a diffusion model. Background Art

[0002] Liver cancer is the sixth most common cancer and the third leading cause of cancer-related death worldwide, with hepatocellular carcinoma (HCC) accounting for 75%-85% of all liver cancer cases. Previous studies have shown that early detection and radical treatment of HCC can significantly improve prognosis, with a 5-year survival rate between 70% and 84%. Semi-annual ultrasonography is widely accepted as a first-line screening method. However, limited image resolution and dependence on operator skills significantly limit its diagnostic efficiency. It has been reported that the combined sensitivity of ultrasonography for early HCC detection is only 0.53. Magnetic resonance imaging (MRI) can provide a high diagnostic sensitivity for HCC due to its diverse soft tissue contrast and high image resolution. However, due to its long “room time” (including patient preparation, image acquisition, and room turnover), high cost, and low accessibility, MRI is mainly recommended for diagnosis rather than a first-line monitoring tool. Therefore, the concept of “abbreviated MRI (aMRI)” (reducing time and cost by reducing pulse sequences) has become a valuable strategy for HCC detection.

[0003] Based on whether gadolinium-based contrast agents (GBCA) are needed, aMRI is mainly divided into two categories: non-contrast aMRI (NC-aMRI) and contrast-enhanced aMRI (CE-aMRI). NC-aMRI does not contain multiphase CE-MRI sequences, significantly shortens the scan time, avoids the use of GBCA, and eliminates potential GBCA-related issues (biosafety risks, additional costs, and inapplicability to patients with allergies and renal insufficiency). However, the most widely used HCC diagnostic guidelines, the Liver Imaging Reporting and Data System (LI-RADS), are based on three main radiological imaging signs: arterial hyper-enhancement (APHE), "washout", and enhancing capsule. These features are derived from multiphase contrast-enhanced scans. Similarly, other important diagnostic guidelines from the American Association for the Study of Liver Diseases (AASLD) and the European Association for the Study of the Liver (EASL) recommend the use of multiphase CE-MRI rather than NC-MRI for the diagnosis of HCC. However, despite preserving important radiological signs, CE-aMRI still requires a long acquisition time and the use of GBCA. In summary, there is currently no aMRI protocol that can ensure high diagnostic accuracy while significantly reducing scan time, eliminating the need for GBCA, and preserving the radiological information of NC-MRI and CE-MRI.

[0004] We hypothesized that reconstructing multi-phase CE-MRI images from NC-MRI using generative deep learning (DL) models may be a valuable solution to simultaneously address the above issues without increasing scan time or using GBCA. Several studies have developed DL models for synthesizing CE-MRI from NC-MRI. However, most of these studies were based on single-center cohorts with limited sample size and population bias. In addition, multi-phase liver CE-MRI has not been attempted to be synthesized. Previous DL algorithms for generating CE-MRI are mainly based on generative adversarial networks (GANs), which are prone to training instability, mode collapse, and authenticity issues of generated samples. Cutting-edge image generation DL algorithms, such as the Stable Diffusion DL model, offer high image quality, flexibility in generated samples, a stable training process, and easy model convergence. However, diffusion DL models have limited applications in medical imaging and have not yet been used for liver CE-MRI image generation. Summary of the invention

[0005] The purpose of the present invention is to propose an algorithm based on a generative artificial intelligence model, which takes pre-enhanced liver MRI images as input, does not require contrast agents, and efficiently and quickly outputs multi-phase (arterial phase, portal venous phase, delayed phase and hepatobiliary specific phase) enhanced images of the liver, and combines the T2WI and DWI actually acquired by MRI scanning to form a simplified liver scanning scheme.

[0006] In order to achieve the above object, the technical solution of the present invention is to provide a simplified liver magnetic resonance imaging method based on generative AI without contrast agent, characterized in that it includes the following steps:

[0007] The first step is to build a stable diffusion model consisting of a vector quantized variational autoencoder and a denoising diffusion probability module. The encoder of the vector quantized variational autoencoder is used to reduce the dimension of the original image to obtain latent space data. The denoising diffusion probability module is then used to achieve image translation in the latent space. Finally, the decoder of the vector quantized variational autoencoder is used to decode the latent space data generated by the denoising diffusion probability module to complete image generation.

[0008] The second step is to train the stable diffusion model constructed in the first step to obtain four independent stable diffusion models, using the pre-enhancement image as the input of the four stable diffusion models, and generating four enhanced late phase images respectively by the four stable diffusion models;

[0009] The third step is to deploy the four stable diffusion models trained in the second step, and use the four stable diffusion models to perform inference based on the pre-enhancement image input in real time.

[0010] Preferably, in the reverse denoising process of the denoising diffusion probability module, the pre-enhancement image and the random noise are tensor-concatenated and then added to the input.

[0011] Preferably, the training of the stable diffusion model includes training of the vector quantized variational autoencoder and training of the denoising diffusion probability module, wherein:

[0012] When training the vector quantized variational autoencoder: the loss function includes the sum of the L1 Loss between the reconstructed image and the real image and the L2 Loss between the codebook and the input image encoded as hidden states, which is optimized by the Adam optimizer;

[0013] When the denoising diffusion probability module is trained: the loss function is the L2 loss of the noise prediction and actual noise output by UNet receiving the target image, the image before enhancement, and t time steps, which is optimized by the Adam optimizer.

[0014] Preferably, the structural similarity index, mean square error, normalized root mean square error and peak signal-to-noise ratio are used to quantitatively evaluate the images generated by the trained stable diffusion model and the actually obtained MRI images.

[0015] The present invention establishes a deep learning model based on the stable diffusion model (SD), takes the pre-enhancement MRI image as input, and uses the enhanced images of each phase actually acquired through MRI scanning as the gold standard training model. After the model training is completed, any pre-enhancement image can be input, so that the model can output four-phase liver enhancement images.

[0016] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0017] 1) No need for contrast agent injection; 2) Based on the 3D neural network architecture, it can efficiently perceive the three-dimensional structure of the image; 3) The generated multi-phase images are spatially registered with each other, which is conducive to disease diagnosis and evaluation; 4) The efficiency of MRI examination of malignant liver lesions is greatly improved; 5) The cost of patients and medical institutions is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The comparison between deep learning-based aMRI (DL-aMRI) and conventional full-sequence MRI (cMRI) is shown;

[0019] Figure 2 Schematic diagram of the model framework, in which: x is the image input (pre), y (AP / VP / TP / HBP) is the generated target, and y′ is the image synthesized by deep learning (AP-syn / VP-syn / TP-syn / HBP-syn), R x , Z y and Z y ′ is the implicit representation of x, y, and y′ given by VQ-VAE, the data dimensions of x, y, and y′ are 1×256×256×32 (channel×height×width×depth), and Z x , Z y and Z y The data dimension of ′ is 3×64×64×8; DWI—diffusion-weighted imaging; Pre—pre-contrast T1-weighted MRI; AP—T1-weighted arterial phase MRI; VP—T1-weighted portal venous phase MRI; TP—T1-weighted transition phase MRI; HBP—T1-weighted hepatobiliary phase MRI; AP, VP, TP, and HBP synthesized by DL are called AP-syn, VP-syn, TP-syn, and HBP-syn, respectively;

[0020] Figure 3The quantitative indicators used to evaluate the quality of CE-MRI images synthesized by deep learning at different enhancement stages are illustrated, including SSIM, NRMSE, MSE and PSNR, where SSIM is the structural similarity index, NRMSE is the normalized root mean square error, MSE is the mean square error, and PSNR is the peak signal-to-noise ratio.

[0021] Figure 4 Representative images of cMRI and DL-aMRI of four patients (AD) confirmed to have HCC are shown, where both cMRI and DL-aMRI include T2, DWI, ADC, and Pre. Yellow arrows indicate HCC lesions. ADC - apparent diffusion coefficient;

[0022] Figure 5 The figure shows the consistency between DL-aMRI and cMRI in HCC size measurement, including the training set, test set, external validation set and the entire data set. In the figure, the solid line is the linear fitting curve, the dotted line is the y=x curve, the slope is the slope of the linear fit, and R2 is the coefficient of determination. DETAILED DESCRIPTION

[0023] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0024] The present invention discloses a stable diffusion DL-based CE-aMRI (DL-aMRI) strategy for detecting HCC without GBCA, while retaining the information of CE-MRI and NC-MRI and significantly reducing the acquisition time. The DL-aMRI protocol proposed in the present invention is based on CE-MRI synthesized by NC-MRI and DL, such as Figure 1 shown.

[0025] Specifically, a simplified liver magnetic resonance imaging method based on generative AI without contrast agent disclosed in an embodiment of the present invention comprises the following steps:

[0026] Step 1: Model design.

[0027] like Figure 2As described, the embodiment of the present invention independently trains four stable diffusion models (Stable Diffusion models) to respectively realize the generation of four enhanced late phases from the pre-enhanced image (Pre-contrast enhanced T1WI, Pre) - post-contrast (Gd-EOB-DTPA) enhanced T1-weighted arterial phase (AP for short), portal venous phase (VP for short), transitional phase (TP for short) and hepatobiliary phase (HBP for short) image models. The architecture of each stable diffusion model is exactly the same, including a vector quantized variational autoencoder (VQ-VAE) and a denoising diffusion probability module (DDPM). The encoder of VQ-VAE mainly reduces the dimensionality of the original image to obtain "compressed" latent space data, while DDPM realizes image translation in the latent space. Finally, the decoder of VQ-VAE can decode the latent space data generated by DDPM to complete image generation.

[0028] The design of the stable diffusion model further includes:

[0029] Step 101: Design of VQ-VAE, including:

[0030] Design of VQ-VAE encoder:

[0031] The encoder contains two encoder blocks (referred to as blocks), which are defined as encoder block EB1 (EncoderBlock 1) and encoder block EB2 (Encoder Block 2). Each block consists of a 3D initial convolution layer and two residual connection blocks (ResNetBlock). The details are as follows:

[0032] For the encoder block EB1, it includes: 3D convolution layer (K4S2, in channel = 1, out channel = 128), residual connection block ResNetBlock1 (including 3D convolution layer (in channel = 128, out channel = 256, K3S1), 3D convolution layer (in channel = 256, out channel = 128), ReLU activation layer), residual connection block ResNetBlock2 (including 3D convolution layer (in channel = 128, out channel = 256), 3D convolution layer (in channel = 256outchannel = 128), ReLU activation layer).

[0033] For the encoder block EB2, it includes: 3D convolution layer (K4S2, in channel = 128, out channel = 256), residual connection block ResNetBlock1 (including 3D convolution layer (in channel = 256, out channel = 256), 3D convolution layer (in channel = 256, out channel = 256), ReLU activation layer), residual connection block ResNetBlock2 (including 3D convolution layer (in channel = 256, out channel = 256), 3D convolution layer (in channel = 256, outchannel = 256), ReLU activation layer).

[0034] Output layer: 3D convolution layer (in channel = 256, out channel = 3, K3S1)

[0035] Design of VQ-VAE decoder:

[0036] The decoder contains two decoder blocks (referred to as blocks), which are defined as decoder block DB1 (DecoderBlock 1) and decoder block DB2 (Decoder Block 2). The details are as follows:

[0037] For the decoder block DB1, it includes: 3D convolution layer (K3S1, in channel = 3, out channel = 256), residual connection block ResNetBlock1 (including 3D convolution layer (in channel = 256, out channel = 256), 3D convolution layer (in channel = 256, out channel = 256), ReLU activation layer), residual connection block ResNetBlock2 (including 3D convolution layer (in channel = 256, out channel = 256), 3D convolution layer (in channel = 256, out channel = 256), ReLU activation layer), transposed convolution layer (K4S2, in channel = 256, out channel = 128).

[0038] For the decoder block DB2, it includes: residual connection block ResNetBlock1 (including 3D convolution layer (in channel = 128, out channel = 256), 3D convolution layer (in channel = 256, out channel = 128), ReLU activation layer), residual connection block ResNetBlock2 (including 3D convolution layer (in channel = 128, out channel = 256), 3D convolution layer (in channel = 256, out channel = 128), ReLU activation layer), transposed convolution layer (K4S2, in channel = 128, out channel = 1).

[0039] Vector Quantization module (VQ-Quantilizer): Referring to the previous literature report https: / / arxiv.org / abs / 1711.00937, the codebook size is 16384 and the encoding dimension is 3.

[0040] Step 102: Design of DDPM:

[0041] DDPM includes a forward denoising process and a reverse denoising process. Since the image generation task of the present invention is essentially image translation, in the reverse denoising process, the pre-enhancement image (T1-PRE) is added to the input as a condition, and is added to the input after tensor splicing with random noise. In an embodiment of the present invention, the forward denoising process is similar to that of previous literature, and the reverse process is based on a 3D U-Net neural network. The DDPM model mainly includes three down-sampling modules (Down SampleBlock 1-3, D1-D3) and three up-sampling modules (Up Sample Block 1-3, U1-3) and Middle Block (MB). The time encoding in the diffusion model is mainly based on the sinusoidal timestep embeddings strategy reported in previous literature, and the time encoding is added to the residual block through a linear transformation layer. The specific model details are as follows:

[0042] DDPM includes:

[0043] 1) The initial convolution layer further includes: a three-dimensional convolution layer (Kernel Size = 3, in channel = 6, out channel = 128).

[0044] 2) Downsampling module D1, further comprising:

[0045] Residual connection block, including: batch normalization layer (Group Normalization Layer), SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 128, out channel = 128);

[0046] Downsampling 3D convolutional layer (Kernel Size = 3, Stride = 2, in channel = 256, out channel = 256);

[0047] 3) Downsampling module D2, further comprising:

[0048] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 128, out channel = 256);

[0049] Multi-head self-attention layer (number of head = 256);

[0050] Downsampling 3D convolutional layer (Kernel Size = 3, Stride = 2, in channel = 256, out channel = 256);

[0051] Attention residual connection block, including: Group Normalization Layer, multi-head self-attention layer (number of head = 512).

[0052] If the number of channels of input and output are inconsistent, the residual connection needs to be performed through the convolutional layer (Kernel Size = 1).

[0053] 4) Downsampling module D3, further comprising:

[0054] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 256, out channel = 512);

[0055] Attention residual connection block, including: Group Normalization Layer, multi-head self-attention layer (number of head = 512).

[0056] If the number of channels of input and output are inconsistent, the residual connection needs to be performed through the convolutional layer (Kernel Size = 1).

[0057] 5) MB, further comprising:

[0058] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 512, out channel = 512);

[0059] Attention residual connection block, including: Group Normalization Layer, multi-head self-attention layer (number of head = 512);

[0060] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 512, out channel = 512).

[0061] 6) Up-sampling module U3, further comprising:

[0062] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 1024, out channel = 512);

[0063] Attention residual connection block, including: Group Normalization Layer, multi-head self-attention layer (number of head = 512);

[0064] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 768, out channel = 512);

[0065] Attention residual connection block, including: Group Normalization Layer, multi-head self-attention layer (number of head = 512);

[0066] The upsampling block includes: double interpolation followed by a 3D convolutional layer (in channel = 512, out channel = 512, Kernel Size = 3).

[0067] If the number of channels of input and output are inconsistent, the residual connection needs to be performed through the convolutional layer (Kernel Size = 1).

[0068] 7) Up-sampling module U2, further comprising:

[0069] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 768, out channel = 256);

[0070] Attention residual connection block, including: Group Normalization Layer, multi-head self-attention layer (number of head = 256);

[0071] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 384, out channel = 256);

[0072] Attention residual connection block, including: Group Normalization Layer, multi-head self-attention layer (number of head = 256);

[0073] The upsampling block includes: double interpolation followed by a 3D convolutional layer (in channel = 256, out channel = 256, Kernel Size = 3).

[0074] If the number of channels of input and output are inconsistent, the residual connection needs to be performed through the convolutional layer (Kernel Size = 1).

[0075] 8) The up-sampling module U1 further comprises:

[0076] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 384, out channel = 128);

[0077] Residual connection block, including: Group Normalization Layer, SiLU activation layer, 3D convolution layer (Kernel Size = 3, in channel = 256, out channel = 128).

[0078] If the number of channels of input and output are inconsistent, the residual connection needs to be performed through the convolutional layer (Kernel Size = 1).

[0079] 9) Output layer, further comprising: Group Normalization Layer, SiLU activation layer, convolutional layer (Kernel Size = 3, in channel = 128, out channel = 3).

[0080] Step 2: Model deployment and training, including:

[0081] Step 2.1, deployment and training of VQ-VAE: The loss function includes the L1 Loss between the reconstructed image and the real image and the sum of the L2 Loss between the codebook and the input image encoded as hidden states. VQ-VAE is optimized by the Adam optimizer with an initial learning rate of 1e-4. The model is built on the Pytorch platform and trained for 500 epochs.

[0082] Step 2.2, DDPM deployment and training: The loss function is the L2 loss between the target image received by UNet, the image before enhancement, the noise prediction output at time step t and the actual noise. DDPM is optimized by Adam optimizer with an initial learning rate of 2.5e-5. A total of 2000 epochs are trained. The model is built on the Pytorch platform and trained for a total of 2000 epochs.

[0083] Step 3: Model performance evaluation.

[0084] The images generated by the deep learning model and the actual MRI images were quantitatively evaluated using the Structural Similarity Index Measure (SSIM), Mean Squared Error (MSE), Normalized Root Mean Square Error (NRMSE), and Peak Signal-to-Noise Ratio (PSNR). Figure 3 As shown, there are four phase image quantitative indicators generated for the training group, validation group, and external validation group. Figure 4 cMRI and DL-aMRI images of four HCC cases are shown. Compared with the real images, the lesions are well displayed in the DL-synthesized MRI. Important imaging features, such as arterial phase enhancement (APHE), washout, and hypointensity in the hepatobiliary phase, are also accurately preserved in the DL-synthesized images. Figure 5 The linear fitting curves in Figure 2 (with tumor size measured by DL-aMRI as the independent variable and tumor size measured by cMRI as the dependent variable) show the consistency of conventional cMRI and DL-aMRI in measuring tumor size. The R2 values ​​obtained by linear fitting ranged from 0.944 to 0.968 in the training set, test set, validation set, and the entire data set.

Claims

1. A simplified liver magnetic resonance imaging method based on generative AI without contrast agent, characterized in that: The following steps are involved: The first step is to build a stable diffusion model consisting of a vector quantized variational autoencoder and a denoising diffusion probability module. The encoder of the vector quantized variational autoencoder is used to reduce the dimension of the original image to obtain latent space data. The denoising diffusion probability module is then used to achieve image translation in the latent space. Finally, the decoder of the vector quantized variational autoencoder is used to decode the latent space data generated by the denoising diffusion probability module to complete image generation. The second step is to train the stable diffusion model constructed in the first step to obtain four independent stable diffusion models, using the pre-enhancement image as the input of the four stable diffusion models, and generating four enhanced late phase images respectively by the four stable diffusion models; The third step is to deploy the four stable diffusion models trained in the second step, and use the four stable diffusion models to perform inference based on the pre-enhancement image input in real time.

2. A simplified liver magnetic resonance imaging method based on generative AI without contrast agent as claimed in claim 1, characterized in that: In the reverse denoising process of the denoising diffusion probability module, the pre-enhancement image and the random noise are tensor-concatenated and added to the input.

3. A simplified liver magnetic resonance imaging method based on generative AI without contrast agent as claimed in claim 1, characterized in that: The training of the stable diffusion model includes the training of the vector quantized variational autoencoder and the training of the denoising diffusion probability module, wherein: When training the vector quantized variational autoencoder: the loss function includes the sum of the L1 Loss between the reconstructed image and the real image and the L2 Loss between the codebook and the input image encoded as hidden states, which is optimized by the Adam optimizer; When the denoising diffusion probability module is trained: the loss function is the L2 loss of the noise prediction and actual noise output by UNet receiving the target image, the image before enhancement, and t time steps, which is optimized by the Adam optimizer.

4. A simplified liver magnetic resonance imaging method based on generative AI without contrast agent as claimed in claim 1, characterized in that: The images generated by the trained stable diffusion model and the actually obtained MRI images were quantitatively evaluated using the structural similarity index, mean square error, normalized root mean square error and peak signal-to-noise ratio.

Images