Deep learning-based conceptualization method for converting T1 weighted image into magnetic resonance angiography and system for converting T1 weighted image into cerebrovascular image

Abstract

The present disclosure provides an AI engine configured as a UNet model. The UNet model includes an encoder and a decoder, which can synthesize a 3D MRA image and its MIP image using a single contrast MR image (T1 weighted MR image) acquired for the same subject, while maintaining continuity of vascular anatomy and important vascular morphological features.

Description

Technical Field

[0001] The present disclosure relates to a conceptualized system and method for implementing conversion of T1-weighted images to magnetic resonance angiography (MRA) based on deep learning. Background Art

[0002] Magnetic resonance angiography (MRA) is a specialized magnetic resonance imaging sequence that utilizes blood flow effects to highlight vascular features and combines maximum intensity projection (MIP) to display the vascular tree.

[0003] One of the existing systems uses a generative adversarial network (GAN) to generate synthetic magnetic resonance angiography (MRA) for data augmentation and anonymization. This deep learning-based method converts conventional single- or multi-contrast MRI images into MRA and requires additional maximum intensity projection (MIP) techniques to obtain better views of organs such as blood vessels, arteries, veins, bronchi, etc. from different angles. A major limitation in generating synthetic MRA from conventional MRI images is the problem of discontinuous vascular anatomy. In diagnosing cerebrovascular diseases such as stroke, timeliness is crucial. For clinicians, it is generally more preferable to use digitally generated cerebrovascular anatomical maps for non-invasive diagnosis.

[0004] When diagnosing cerebrovascular diseases, there is a tendency to perform timely and non-invasive MRI diagnosis in a faster and cheaper manner. Therefore, magnetic resonance imaging is an excellent technique for diagnosing cerebrovascular diseases. There will be significant advantages in minimizing the magnetic resonance scanning time when generating specific rare imaging modalities (such as MRA images) that highlight vascular anatomical details. However, it is difficult for the prior art to achieve this goal because existing systems need to rely on exogenous or endogenous contrast mechanisms and require the use of specialized imaging sequences. Summary of the Invention

[0005] According to a first aspect of the present disclosure, there is provided a method for a system to synthesize an MRA image and its MIP image using the acquired single-contrast MR image (T1-weighted MR image). The system has at least a processor and a memory therein to execute instructions of an artificial intelligence engine, and the artificial intelligence engine is configured with a UNet model stored in the memory of the system; wherein, the UNet model includes:

[0006] An encoder having a plurality of layer blocks, each layer block of the encoder including one or more convolutional layers, each of the convolutional layers being associated with an activation layer and a downsampling layer;

[0007] A decoder having a plurality of layer blocks, each layer block of the decoder including an upsampling layer and one or more convolutional layers, each of the convolutional layers being associated with an activation layer;

[0008] Skip connections for associating one of the layer blocks of the encoder with one of the layer blocks of the decoder at corresponding multi-scale resolution levels;

[0009] wherein, the encoder is used to extract features from the T1-weighted MR image for the decoder to combine the output from the encoder and the extracted image features at multi-scale resolution levels through the skip connections to generate an MRA image and its MIP image.

[0010] According to the first aspect, the decoder includes an output layer for generating an image having the same resolution as the input image.

[0011] According to the first aspect, the output layer includes a single output convolutional layer followed by an output activation layer.

[0012] According to the first aspect, the single output convolutional layer is a 1×1 convolutional layer with a stride of 1.

[0013] According to the first aspect, the output activation layer is used to perform a hyperbolic tangent (tanh) operation.

[0014] According to the first aspect, the encoder and the decoder are used to execute a cross-sequence for converting from a T1-weighted image to an MRA or MIP image composed of 19 convolutional layers.

[0015] According to the first aspect, the encoder is used to receive an image including three dimensions and one or more color channels.

[0016] According to the first aspect, one or more layer blocks of the encoder include repeating two 3x3 convolutional layers in five layer blocks with a stride of 2 voxels.

[0017] According to the first aspect, the layer block of the encoder immediately before the decoder includes a single convolutional layer.

[0018] According to the first aspect, zero-padding technology is implemented before each convolutional layer.

[0019] According to the first aspect, the activation layer is used to perform a linear rectification function through one or more rectified linear units (ReLU).

[0020] According to the first aspect, the downsampling includes a 2×2×2 max pooling operation with a stride of 2 voxels.

[0021] According to the first aspect, each convolutional layer is used to process input data using multiple convolutional filters.

[0022] According to the first aspect, in the encoder, from the first layer block to the last layer block, the number of convolutional filters doubles.

[0023] According to the first aspect, the upsampling layer of the decoder is used to perform nearest neighbor interpolation to increase the size of the image through each layer block within the decoder.

[0024] According to the first aspect, one or more convolutional layers of the decoder use random initialization and unequal kernel sizes.

[0025] According to the first aspect, the skip connections are used to copy and concatenate features generated from one of the layer blocks of the encoder to one of the layer blocks of the decoder at corresponding multi-scale resolution levels.

[0026] According to the first aspect, the training batch size used by the UNet model is 4.

[0027] According to the first aspect, the UNet model is trained using consecutive MRA images that vary in the same way as the input image.

[0028] According to the first aspect, the UNet model is trained for more than 100 training epochs.

[0029] According to a second aspect of the present disclosure, there is provided a system for converting a T1-weighted image into an image of the vasculature of the brain based on deep learning. The system includes: a memory for storing instructions and a processor for executing the instructions stored in the memory; the processor executes an artificial intelligence engine, and the artificial intelligence engine is configured with a UNet model stored in the memory of the system; wherein, the UNet model includes:

[0030] An encoder having a plurality of layer blocks, each layer block of the encoder includes one or more convolutional layers, and each of the convolutional layers is associated with an activation layer and a downsampling layer;

[0031] A decoder having a plurality of layer blocks, each layer block of the decoder includes an upsampling layer and one or more convolutional layers, and each of the convolutional layers is associated with an activation layer;

[0032] Skip connections for associating one of the layer blocks of the encoder with one of the layer blocks of the decoder at corresponding multi-scale resolution levels;

[0033] Wherein, the encoder is used to extract features from the T1-weighted MR image for the decoder to combine the output from the encoder and the extracted image features at multi-scale resolution levels through the skip connections to generate an MRA image and its MIP image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Embodiments of the present disclosure will be described below by way of example with reference to the accompanying drawings.

[0035] Figure 1 Schematic diagram of a computer server according to an embodiment of the present disclosure, the computer server being used to implement a system for synthesizing MRI-MIP from MRI data.

[0036] Figure 2 Block diagram showing the process for registering T1-weighted to MRA images.

[0037] Figure 3 Block diagram showing the process for converting T1-weighted to MRA to MIP images with two paths.

[0038] Figure 4 Schematic diagram of the encoder-decoder UNET architecture for image conversion in an embodiment of the present disclosure.

[0039] Figure 5 Shows various images, including (a) illustration showing T1-weighted and MRA images for an experiment of an embodiment of the present disclosure; (b) illustration showing synthesized MRA and corresponding difference maps for a batch size of 1 in the experiment of an embodiment of the present disclosure; (c) illustration showing synthesized MRA and corresponding difference maps for a batch size of 2 in the experiment of an embodiment of the present disclosure; (d) illustration showing synthesized MRA and corresponding difference maps for a batch size of 4 in the experiment of an embodiment of the present disclosure; (e) illustration showing synthesized MRA and corresponding difference maps for a batch size of 1 in the experiment of an embodiment of the present disclosure; (f) illustration showing graphical representation of peak signal-to-noise ratio (PSNR) of synthesized MRA images from (a) to (e); (g) illustration showing graphical representation of structural similarity index (SSIM) of synthesized MRA images from (a) to (e).

[0040] Figure 6 Shows various images, including (a) illustration showing T1-weighted and MIP images for an experiment of an embodiment of the present disclosure; (b) illustration showing synthesized MIP and corresponding difference maps for a batch size of 1 in the experiment of an embodiment of the present disclosure; (c) illustration showing synthesized MIP and corresponding difference maps for a batch size of 2 in the experiment of an embodiment of the present disclosure; (d) illustration showing synthesized MIP and corresponding difference maps for a batch size of 4 in the experiment of an embodiment of the present disclosure; (e) illustration showing synthesized MIP and corresponding difference maps for a batch size of 1 in the experiment of an embodiment of the present disclosure; (f) illustration showing graphical representation of peak signal-to-noise ratio (PSNR) of synthesized MIP images from (a) to (e); (g) illustration showing graphical representation of structural similarity index (SSIM) of synthesized MIP images from (a) to (e).

[0041] Figure 7A set of visualization diagrams of the three - subject test including the proposed model, including (a) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MRA, showing a 2D axial middle slice of the T1 - weighted image; (b) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MRA, showing a 2D axial middle slice of the ground - truth MRA image; (c) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MRA, showing a 2D axial middle slice of the synthetic MRA image; (d) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MRA, showing a 2D axial middle slice of the difference map between the synthetic MRA and the ground - truth MRA.

[0042] Figure 8 A set of visualizations for the three - subject test of the proposed model, including (a) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MIP and MRA MIP, showing a 2D axial middle slice of the T1 - weighted image; (b) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MIP and MRA MIP, showing a 2D axial middle slice of the ground - truth MIP image; (c) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MIP and MRA MIP, showing a 2D axial middle slice of the synthetic MIP image; (d) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MIP and MRA MIP, showing a 2D axial middle slice of the difference map between the synthetic and ground - truth MIP images; (e) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MIP and MRA MIP, showing a 2D axial middle slice of the synthetic MRA / MIP image; (f) a set of visualization diagrams of the three - subject test of the proposed model on the translation from T1 - weighted to MIP and MRA MIP, showing a 2D axial middle slice of the difference map between the synthetic MRA / MIP and the ground - truth MIP; (g) a set of visualization diagrams of a box plot comparing the average PSNR between MIP and MRA / MIP; (h) a set of visualization diagrams of a box plot comparing the average SSIM between MIP and MRA / MIP.

[0043] Figure 9A set of visualization diagrams, including (a) a set of visualization diagrams of T1-weighted images and corresponding ground truth MIP images according to an embodiment of the present disclosure; (b) a set of visualization diagrams of MRA / MIP images and difference maps after 100 training cycles according to an embodiment of the present disclosure; (c) a set of visualization diagrams of MRA / MIP images and difference maps after 1000 training cycles according to an embodiment of the present disclosure; (d) a set of visualization diagrams of MIP images and difference maps after 100 training cycles according to an embodiment of the present disclosure; (e) a set of visualization diagrams of MIP images and difference maps after 100 training cycles according to an embodiment of the present disclosure.

[0044] Figure 10 Shows a set of diagrams, including (a) a diagram showing a box plot comparing PSNR between MRA / MIP after 100 training cycles and 1000 training cycles; (b) a diagram showing a box plot comparing PSNR between MIPs after 100 training cycles and 1000 training cycles; (c) a diagram showing a box plot comparing PSNR between MRA / MIP and MIP after 1000 training cycles; (d) a diagram showing a box plot comparing SSIM between MRA / MIP after 100 training cycles and 1000 training cycles; (e) a diagram showing a box plot comparing SSIM between MIPs after 100 training cycles and 1000 training cycles; (f) a diagram showing a box plot comparing (c) SSIM between MRA / MIP and MIP after 1000 training cycles.

[0045] Figure 11 Is a line graph, including (a) a line graph showing the detection accuracy of cross-modal image detection, where the model is trained on TCG and tested on GAN; (b) a diagram showing the detection accuracy of cross-modal image detection, where the model is trained on TCG and tested on DM; (c) a diagram showing the detection accuracy of cross-modal image detection, where the model is trained on DM and tested on GAN; (d) a graph showing the detection accuracy of cross-modal image detection, where the model is trained on GAN and tested on DM.

[0046] Figure 12 Is a block diagram showing the workflow steps and components of a method according to an embodiment of the present disclosure including model training and validation.

[0047] Figure 13 Is a block diagram illustration showing the routine steps of a method according to an embodiment of the present disclosure for generating a simulated MIP image from a T1-weighted image. Detailed Description

[0048] Embodiments of the present disclosure provide a computerized system or method that uses a novel deep learning process to synthesize 3D maximum intensity projections (MIPs) of magnetic resonance imaging (MRI) images using a single acquired T1-weighted (T1-w) MRI image. In one embodiment of the present disclosure, a computer system 100 for performing a method including an artificial engine is provided, the artificial engine having a multi-block UNet model (or referred to as a "model") 120 implemented with an L2 loss function for encoding and decoding image data between each block, emphasizing and minimizing the statistical feature differences between the target and the 3D volume of the synthetic image.

[0049] In this embodiment, the UNET model 120 includes an encoder 122 and a decoder 124, which can use a single contrast MR image (T1-weighted MR image) acquired for the same object to synthesize a 3D MRA image and its MIP image while maintaining the continuity of vascular anatomy and important vascular morphological features. In particular, the disclosure aims to address methods of directly converting from T1-weighted images to MIPs (such as Figure 2 and Figure 3 shown), evaluate them, and compare them with existing methods of converting from T1-weighted images via MRA to MIP. The present disclosure paves the way for easily creating digital MIP twins without worrying about the generation of defects in the vascular structure.

[0050] As Figure 1 shown, a schematic diagram of a computer system or computer server 100 is shown, the system being implemented as a system for using a configuration Figure 4An exemplary implementation of a system for synthesizing MRA / MIP from one or more MRI images by the AI engine of the UNet model shown. In one embodiment of the present disclosure, the system includes a server 100 that includes relevant components required to receive, store, and execute appropriate computer instructions. These components may include: a processing unit 102 that includes a Central Processing Unit (CPU), a Math Co-processing Unit (MathProcessor), a Graphics Processing Unit (GPU), or a Tensor Processing Unit (TPU) for tensor or multi-dimensional array calculation or manipulation operations; a Read-Only Memory (ROM) 104; a Random-Access Memory (RAM) 106; an Input / Output (I / O) device (e.g., a disk drive 108); an input device 110 (e.g., an Ethernet port, a USB port, etc.); a display 112, e.g., a liquid crystal display, a light-emitting display, or any other suitable display; and a communication link 114. The server 100 may include instructions that can be stored in the ROM 104, the RAM 106, or the disk drive 108 and executed by the processing unit 102. Multiple communication links 114 may be provided, which may be connected to one or more computing devices, such as servers, personal computers, terminals, wireless or handheld computing devices, and edge computing devices, in different ways. At least one of the multiple communication links may be connected to an external computing network via a telephone line or other type of communication link.

[0051] The server 100 may include storage devices such as a disk drive 108, and the disk drive 108 may include a solid-state drive, a hard disk drive, an optical drive, a tape drive, or a remote or cloud-based storage device. The server 100 may use a single disk drive or multiple disk drives, or a remote storage service 120. The server 100 may also have an appropriate operating system 116 residing on the disk drive or in the ROM of the server 100.

[0052] A computer or computing device can also provide the necessary computing power to operate or configure the AI engine docked to the Unet model 120. The AI engine can be implemented locally or can also be accessed or partially accessed through a server or cloud-based service. In one embodiment of the present disclosure, the system 100 is implemented as a 64-bit Linux system with an Intel core and 164GB of RAM, using the Kera API with TensorFlow (version 2.8) as the backend in Python (version 3.9); although, as understood by those skilled in the art, alternative hardware can also be used.

[0053] In one embodiment, a system 100 is provided. The system 100 includes memories 104, 106 for storing instructions; and a processor or processing unit 102 for executing the instructions stored in the memories. The processing unit or processor 102 is used to execute an artificial intelligence engine that is configured as the UNet model 120 stored in the memories 104, 106.

[0054] The UNet model 120 includes an encoder 122 having a plurality of layer blocks, a decoder 124 having a plurality of layer blocks, and skip connections 126. Each of the layer blocks in the encoder 122 includes one or more convolutional layers and a downsampling layer. Each convolutional layer is associated with an activation layer. Each layer block of the decoder includes an upsampling layer and one or more convolutional layers, and each convolutional layer is associated with an activation layer. Each skip connection 126 is associated with one of the layer blocks in the encoder 122, wherein one of the layer blocks of the decoder 124 is at a corresponding multi-scale resolution level. The encoder 122 is used to extract a larger dataset of low to high-level features in the multi-scale resolution levels from the T1-weighted MR image for the decoder 124 to combine the output from the encoder and the extracted image features in the multi-scale resolution levels through the skip connections to generate an MRA image and its MIP image. In a preferred embodiment, a batch normalization operation is performed before passing the data from the convolutional layer to the activation layer.

[0055] The encoder is used to receive an image including three-dimensional voxels and one or more color channels. In one embodiment, one or more of the layer blocks of the encoder include two 3x3 convolutional layers repeatedly implemented in five layer blocks with a stride of 2 voxels. The last layer block of the encoder 122 immediately before the decoder 124 includes a single convolutional layer. Zero-padding is performed before each convolutional layer. The activation layer is used to perform a linear rectification function through one or more rectified linear units (ReLU). In one embodiment, the downsampling includes a 2×2×2 max pooling operation with a stride of 2 voxels. Each convolutional layer is used to process the input data with a plurality of convolutional filters, and the number of convolutional filters doubles from the first layer block to the last layer block.

[0056] The encoder 122 and the decoder 124 are used to perform a cross-sequence conversion from a T1-weighted image to an MRA or MIP image consisting of 19 convolutional layers.

[0057] The output includes a single 1×1 output convolutional layer with a stride of 1, followed by an output activation layer with a hyperbolic tangent (tanh) operation to generate an image with the same resolution as the input image.

[0058] The upsampling layer of the decoder is used to perform nearest neighbor interpolation to increase the image size through each layer block within the decoder. One or more convolutional layers of the decoder use random initialization and unequal kernel sizes.

[0059] The skip connection 126 is used to copy and concatenate the features generated from one of the layer blocks of the encoder to one of the layer blocks of the decoder at the corresponding multi-scale resolution level.

[0060] Preferably, the UNet model 120 in the embodiments of the present disclosure is preferably trained with a batch size of 4, and the training data is continuous MRA images, which are different in the same manner as the input images, and the number of training epochs exceeds 100.

[0061] Magnetic resonance imaging (MRI) has become one of the most powerful diagnostic tools for evaluating and examining patients with cerebrovascular diseases in radiology clinics. The advantage of MRI is its ability to provide cross-sectional images of anatomical regions in any plane and with good soft tissue contrast.

[0062] The principle of MRI is based on the basic physics of nuclear magnetic resonance imaging (NMR), by utilizing the nuclear spin energy level transitions of hydrogen atoms in water and fat in human tissues. The patient movement and the need for a long scanning time.

[0063] Although MRA is crucial for the rapid diagnosis of the second leading cause of disability and death worldwide, there are still some practical and clinical problems, including the possible need to use contrast agents for patients, increased acquisition and processing times, and diagnostic costs. Therefore, some doctors may be reluctant to recommend or order an MRA examination unless there is a strong necessity for retrospective examination of vascular structures or dealing with intravascular problems.

[0064] The rapid development of artificial intelligence (AI) and its popularization to more users enable an embodiment of the present disclosure to synthesize cross-modal and intra-modal medical images through deep learning algorithms. An image-to-image conversion method has been adopted to generate synthetic MRA images from each other single image modality. Another method has been adopted to generate synthetic MRA images from multi-input applications, including major structural imaging sequences (T1-weighted and T2-weighted MRI), or T1, T2, PD-weighted images. However, these existing systems rely on generative adversarial networks (GANs) with high computational overhead. Some systems use U-Net AI models for image segmentation, which can extract local and global features from input images but are not designed for synthesizing MRA images.

[0065] For existing MRA image synthesis systems, the target output is an MRA image, which requires the use of the maximum intensity projection (MIP) algorithm to display vascular data of interest through logic and probability. This technique is called MIP-MRA. MIP-MRA is crucial for performing topological analysis of cerebral blood vessels. Existing MRA / MIP-MRA image synthesis systems have some problems when synthesizing images, such as disconnection of the tree structure caused by artificial gaps introduced by a limited image slice sequence, artifacts generated by overlapping during the conversion process, and sensitivity to changes in the contrast-to-noise ratio of the input images, although current research efforts have focused on developing problem-free algorithms.

[0066] The present disclosure provides a novel AI engine for early processing and generating virtual vascular MRA images from conventional MRI scan images. The AI engine of an embodiment of the present disclosure is configured to include a UNET-based model for synthesizing MIP-MRA images from T1-weighted MRI contrast sequences. In this way, the present disclosure can directly convert conventional MRI contrast images into MIP-MRA image sequences.

[0067] Considering the clinical impact of vascular problems and the time sensitivity in diagnosing cardiovascular diseases (CVD), the present disclosure provides a system or method for efficiently generating visual displays to assist clinicians in early detection, which is very advantageous. The present disclosure also has the following advantages, that is, it provides a system or method that can reduce clinical time waste by avoiding (i) multi-step processing of MRA images and (ii) multi-contrast MRI image acquisition for obtaining MRA images.

[0068] Model architecture

[0069] In another embodiment of the present disclosure, a method is provided. The method includes implementing a UNet model 120 stored in the memory of a computing system 100. The UNet model 120 includes an encoder 122 and a decoder 124 having skip connections 126. The encoder 122 downsamples an input image to extract low- to high-level features of a larger data set, while the decoder 124 combines the output from the encoder and the extracted image features at multi-scale resolution levels to generate an MRA-based target image output through an upsampling process. Skip connections 126 are added between the reflection layers in the encoder 122 and decoder 124 networks to accelerate the information transfer between the input and output 3D image streams. This helps to learn the matching features of the corresponding mirror layers.

[0070] Reference Figure 4 , an embodiment of the UNet model 120 implemented in an embodiment of the present disclosure is shown. The model is used to perform cross-sequence image conversion from T1-weighted (T1-w) to MRA or MIP images and includes an odd number of convolutional layers. In another embodiment of the present disclosure, the UNet model 120 may include 19 convolutional layers.

[0071] In an embodiment of the present disclosure, the input image is normalized to 128×128×64 voxels and is single-channel (grayscale image). However, the UNet model 120 can be used to process input images at higher resolutions. The encoder 122 consists of five layer blocks, where each layer block repeats the implementation of a convolutional layer with a 3×3 kernel for two layers, with a stride of 2 voxels, and the last layer block has only one convolutional layer. Before convolution, zero-padding is used to keep the resolution of the extracted deeper feature maps matching the resolution of the input feature maps. The layer blocks in the encoder 122 may include a first convolutional layer, followed by a rectified linear unit (ReLU) activation layer and a 2×2×2 max pooling operation with a stride of 2. Alternatively, the layer blocks in the encoder 122 may include two or more convolutional layers, where each convolutional layer is followed by a rectified linear unit (ReLU) activation layer and a 2×2×2 max pooling operation with a stride of 2. The advantage of using the ReLU non-linear transfer function between hidden convolutional layers lies in its computational simplicity and representational sparsity, which provides better problem-solving ability and avoids the problem of vanishing gradients.

[0072] In the UNet model 120 of the embodiment of the present disclosure, the max pooling operation implemented after the activation layer is designed to reduce the spatial size of the image feature map to half of the original, thereby reducing the computational cost and saving memory. The number of convolutional filters increases from 16 in the first block to 1024 in the last block. This enables the network to learn the hierarchical relationships of larger receptive fields on MR images.

[0073] The decoder 124 in the embodiments of the present disclosure is configured to be a reverse version of the encoder 122. One difference between the encoder 122 and the decoder 124 is that the max pooling operation in the encoder is replaced by an upsampling operation in the decoder, where nearest neighbor interpolation doubles the image size through each layer block. Since deconvolution uses random initialization and unequal convolution kernel sizes, which can lead to checkerboard artifacts, upsampling is used instead of deconvolution. In addition, the encoder 122 is connected to the decoder 124 through skip connections 124 at multiple scale resolutions to help reconstruct the original spatial resolution level, thereby restoring the original spatial resolution of the input T1-weighted image at the output. The features extracted in each layer block in the encoder 122 are copied and concatenated with the corresponding features in the decoder 124. These concatenations enable the high-level and low-level features in the encoding part to be used as additional inputs in the decoding part, thus providing an effective and stable image representation. The output layer of the UNet in one embodiment of the present disclosure includes a 1x1 convolutional layer with a stride of 1, followed by a hyperbolic tangent (tanh) activation function, which has been proven to provide good results. The final layer reconstructs an output image of the same size as the input image (128×128×64) from a 16-dimensional feature map vector.

[0074] Preprocessing

[0075] Before the image data is input into the AI engine of the embodiments of the present disclosure, the 3D-T1-weighted image is first re-sliced into the axial plane, thereby changing its voxel size. To place the T1-weighted image and its corresponding MRA image in the same spatial coordinate system, a 3D affine registration method is adopted, which includes translation, centroid, rigid body, and full affine registration for registering the 3D volume of the T1-weighted image with the MRA image.

[0076] In one embodiment of the present disclosure, this process is executed through the DIPY (Diffusion Imaging in Python) software library. An example of this process is as Figure 2 shown. In this example, the MRA image and the T1-weighted image are labeled as the static image and the moving image, respectively. To obtain the maximum intensity projection (MIP)-MRA image from the MRA image, the AI engine executed by the system 100 uses a publicly available software library, namely the SimpleITK image analysis library, which can be found on GitHub (https: / / github.com / ljpadam / maximum_intensity_projection), to process the MRA image into a 3D-MIP image. The T1-weighted, MRA, and MIP images are further normalized to obtain voxel intensities between 0 and 1, and are resized to the same matrix size (W×H×D) = 128×128×64 using the spline interpolation scaling (SIZ) method.

[0077] Experimental dataset

[0078] In one embodiment of the present disclosure, system 100 uses the IXI dataset for training and testing, which is publicly available online (https: / / brain-development.org / ixi-dataset / ). The IXI dataset contains different MRI image modalities, including T1-weighted, T2-weighted, PD-weighted, DWI, and MRA images of normal healthy subjects. In the experiments for training and testing the embodiments of the present disclosure, T1-weighted and MRA image modalities were obtained. The dataset containing 522 subjects with paired T1-weighted and MRA images includes 180, 316, and 26 datasets from Hammersmith Hospital (HH), Guys Hospital (Guy), and the Institute of Psychiatry (IOP), respectively. In addition to the multi-institutional acquisition of the dataset, two types of scanners (1.5 and 3 Tesla) and various imaging protocols were used to acquire the images. Notably, the T1-weighted images were acquired in the sagittal plane, and the MRA images were acquired in the axial plane with different slice thicknesses. In addition, all other institutions have images with a size of 512×512 pixels and 100 slices, except for the IOP-based images with a size of 1024×1024 pixels and 92 slices. The data was randomly divided into 70, 20, and 10% for the training, validation, and test sets.

[0079] Model training

[0080] In one embodiment of the present disclosure, the AI engine is configured with a UNet model 120, which uses 512 3D images to train the dataset for generating T1-weighted, MRA, and MIP images after preprocessing. The model trainable parameters are initialized using the uniform distribution technique. In a deep model with ReLU layers, the uniform distribution technique performs better than the Xavier technique. In one embodiment, an adaptive optimizer such as the Adam stochastic optimization algorithm with a learning rate of 0.002 is applied to minimize the mean squared error (MSE) loss function or the L2 loss function in a step-by-step manner, and the trainable network is updated step by step at each training step until the model converges. Mean squared error (MSE) is used as the cost function in training because it has a small computational overhead and can produce a convex optimization problem with a stable gradient.

[0081] In the present disclosure, an end-to-end 3D U-Net architecture model 100 is used for the conversion from T1-weighted images to MRA images, and it contains a total of 2 million trainable parameters. These parameters are optimized through the training dataset during the model training process to learn how to map the source T1-weighted images to the target MRA or MIP contrast, as Figure 5as shown. Refer to Figure 4 , the 3D U-Net architecture model 120 was trained to perform a possible transformation, that is, to convert the T1-weighted image to MRA before obtaining the MIP, so as to reveal the vascular tree. In addition, using the disclosed embodiments, the same model was trained to synthesize MIP directly from the T1-weighted image. Refer to Figure 5 and Figure 6 , Figure 5 and Figure 6 show the experimental results of training and testing a disclosed embodiment. The batch size (i.e., the number of samples per gradient update) was set to 1, 2, 4, and 8 to study which batch size could provide the best performance. During this process, the model was trained for 100 epochs. In addition, it was also studied to use 10 times the initial number of epochs to establish the model performance, and the performance of the model between the underfitting and overfitting stages was analyzed.

[0082] Model Evaluation

[0083] In the experiments conducted using the system 100 of the disclosed embodiments, the residual or difference map was calculated by the formula (x i , y i ) = y i - x i to compare the synthesized MRA and MIP images and perform visual comparison. The quality of the synthesized MRA and MIP images was evaluated by the U-Net with the ground truth images, and two voxel-level metrics were adopted. These metrics were constructed from pixel-level metrics across the image depth: peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) metrics. These metrics are widely used to evaluate the overall quality while capturing the features dominated by lower spatial frequencies, considering both quantitative and qualitative differences and simulating human perception. PSNR, which considers both the mean squared error (MSE) of the image and the highest possible intensity value, is defined as wherein, is the maximum pixel value of the image depending on the data type. SSIM captures the human-perceived quality of the synthesized image by comparing two images. Its formula is given as:

[0084]

[0085] wherein, μ represents the average image intensity, σ 2 represents the variance of the image, σ xy represents the covariance of the ground truth (x) and predicted (y) images, and C1 and C2 represent constants added to stabilize the division with a weak denominator. However, from the experiments conducted using the system 100 of the disclosed embodiments, the mean ± SEM of PSNR and SSIM on all validation datasets was observed.

[0086] Effect of Batch Size on Model Performance

[0087] In Figure 5 (a - e), the results obtained by training the proposed model using different batch sizes in the conversion from T1 - weighted images to MRA images are shown. Using the difference map, the 2D axial slices of the synthetic MRA are compared with the corresponding ground - truth MRA slices. However, changes in batch size are observed with respect to two significant features, including micro - vascular features (white dots) and image contrast. Compared with other batch sizes, the MRA images obtained from batch size 8 do not show vascular features. Quantitative analysis shows that the PSNR ( Figure 5 (f)) and SSIM ( Figure 5 (g)) of the synthetic MRA vary with the change of batch size. However, batch size 4 shows the best results with the highest PSNR and SSIM. The batch sizes with the lowest PSNR and SSIM are 8 and 2 respectively. Therefore, when training a model for converting T1 - weighted images to MRA, image batch size 4 remains the best in obtaining excellent results.

[0088] Examine and compare the visual appearance of the 2D axial slices obtained from the synthetic MIP and the real MIP with different batch sizes to determine the optimized batch size for converting T1w to MIP using the UNet model 120 of the embodiments of the present disclosure. Figure 6 (a - e) show the difference map and MIP images. At low resolution, a close similarity of the vascular tree is observed. However, the sharpness and continuity of the vascular tree vary with the batch size.

[0089] Compared with other batch sizes, the batch size of the image with the worst sharpness is 8. In addition, quantitative analysis is carried out using PSNR ( Figure 6 (f)) and SSIM ( Figure 6 (g)) to compare the differences between the synthetic MIP and the real MIP. It is found that the order of change of PSNR at different batch sizes is the same as that of SSIM, that is: batch size 1 < 8 < 2 < 4. Batch size 4 produces the best results with the highest PSNR and SSIM, while the lowest values are observed for batch size 1. Therefore, when converting T1 - weighted images to MIP and MRA, image batch size 4 remains the best choice for training the proposed model.

[0090] In an embodiment of the present disclosure, when the UNet model 120 converts T1 - weighted images to MRA images, the batch size is set to 4 and is applicable to multiple subjects. Now refer to Figure 8 . For the T1 - weighted images ( Figure 8 (a)) and real MRA images ( Figure 8 (b)) from three different subjects and the synthetic MRA images (Figure 8 (c)) for comparison to obtain a difference map ( Figure 8 (d)). The changes in the T1-weighted image are preserved in the synthesized MRA image and show changes in the microvascular features. Thus, the MRA synthesis of the model is sensitive to anatomical changes in the T1-weighted image, and more vascular features are observed in Subject 3.

[0091] In one embodiment of the present disclosure, when the UNet model 120 converts a T1-weighted image into a MIP image, the batch size is set to 4 and is applicable to multiple subjects. Using the T1-weighted ( Figure 8 (a)) and ground truth MIP ( Figure 8 (b)) and the synthesized MIP ( Figure 8 (c)) and MRA / MIP images ( Figure 8 (e)) for comparison to obtain their respective difference maps shown in Figure 8 (d) and (f) respectively. In the synthesized MIP and MRA / MIP, anatomical changes that produce structural changes in the vascular tree and vascular features can be observed. Thus, the synthesis ability of the model is sensitive to anatomical changes in the T1-weighted image. In addition, more vascular features appear in the synthesized MIP than in the MRA / MIP. In addition, quantitative analysis shows that the synthesized MIP has statistically significantly higher PSNR and SSIM than the MRA / MIP, as shown in Figure 8 (g) and (h).

[0092] Effect of extended training steps on the performance of the proposed model

[0093] Using the MRI scanned from a single object, the system 100 of the present disclosure is used to synthesize MIP and MRA / MIP with the vascular trees at different depths, i.e., the top, middle, and bottom axial slices, as shown in the T1-weighted image of Figure 9 . The UNet model of the present disclosure is trained for 1000 training cycles, and the resulting images are compared with those obtained at 100 training cycles. Depth analysis shows that after 1000 training cycles, additional features of the vascular tree are added to the vascular trees of the synthesized MIP and MRA / MIP. Specifically, the well-developed top and middle slices are maintained, while the degenerated features in the bottom slice are well vascularized in the synthesized MIP image. In addition, on the middle and bottom slices of the MRA / MIP, a slight improvement in the vascular tree is observed. However, the features of the MIP are more than those of the MRA / MIP. Thus, training the model for a longer training cycle improves the vascular tree. In addition, the vascular sharpness is enhanced, but vascular differences, including vascular gaps, are shown in the difference map.

[0094] InFigure 10 (a - f), box plots show the quantification involving PSNR and SSIM that describes the effect of the extended training steps. After 100 training epochs, the observed metric changes include (i) an enhancement of the PSNR of MRA / MIP, (iii) a statistically significant improvement in the SSIM of MRA / MIP, and (ii) a decrease in the PSNR and SSIM of the synthetic MIP. Additionally, there is a statistically significant decrease in both PSNR and SSIM between MRA / MIP and MIP after 100 training epochs. Thus, the longer training steps improve the PSNR of MRA / MIP but are not sufficient to eliminate the difference from the PSNR of MIP, and they also have a negative impact on the SSIM of MIP. For our direct translation method, further research was conducted by increasing the voxel size of the image to 256×256×64. Compared with the 128×128×64 voxels, the MIP obtained after the voxel increment ( Figure 10 (a - b)) shows improved image quality and more improved vascular anatomy.

[0095] From the perspective of synthesizing magnetic resonance images, the system 100 of an embodiment of the present disclosure includes a processor for executing an AI engine configured as a UNet model 120 to synthesize a vascular anatomy tree from a conventional T1 - weighted image scan. The present disclosure reduces the need for multi - contrast MR images to acquire MRA images because reducing the MR contrast images to the earliest acquired images in an MRI scan can also help reduce the time target. Additionally, the system 100 is designed to re - envision the developed synthetic MRA image before obtaining the vascular tree by maximum intensity projection. Moreover, the UNet 120 of an embodiment of the present disclosure is suitable for using methods that employ stable models with fewer tunable parameters and computational costs.

[0096] In one embodiment of the present disclosure, it is recommended to use an optimal batch size of 4 for real - world implementation. The sensitivity of the UNet architecture model for synthesizing MRA and MIP from T1 - weighted images of subjects showing different anatomies at the same axial slice position, e.g., consecutive MRA images that are different in the same way as the input images. Additionally, the obtained and synthetic MIPs show different root structures of the vascular tree.

[0097] The experimental results demonstrate the ability of the UNet model 120 to directly predict vascular features from 3D T1 - weighted images. Visual comparison of the vascular images shows that the synthetic MIP is similar to the real MIP. Due to the low - resolution input used to train the system 100, this similarity decreases as the vascular diameter decreases. However, after training the model for 100 training epochs, it was observed that the vascular tree in MRA / MIP was not fully developed. This indicates that longer training steps will be required, or perhaps a more complex architecture.

[0098] When testing an embodiment of the disclosure, the vascular tree obtained by MRA / MIP was compared with the vascular tree obtained by direct MIP. It was observed that multiple features such as image contrast, brightness, and the presence of small vessel characteristics are necessary in generating the vascular tree. The vascular tree is one of the necessary factors for training the embodiments of the disclosure. Nevertheless, after training the model for 1000 training cycles, it was observed that many smaller vessels were missing in the vascular tree of MRA / MIP compared to the synthetic MIP. This supports the need for a more complex AI model or longer training steps to improve the efficiency of the embodiments of the disclosure. Figures 5 to 10 The average PSNR value of the comparison in [ ] also supports this.

[0099] The experimental results, together with the quantitative analysis of the embodiments of the disclosure, show that the disclosure is capable of identifying more branches of smaller vessels in the vascular tree than existing methods. The evaluation of the disclosure reveals that the quality of the vasculature generated by the embodiments of the disclosure is sensitive to anatomical variations and the quality of T1-weighted images. Therefore, the disclosure provides the potential to eliminate the bottlenecks involved in generating a digital twin of the MIP version of MRA images from one or more structural MRI images, while reducing the patient waiting time for diagnosis.

[0100] In one embodiment of the disclosure, a system 10 is provided, which relates to the maximum intensity projection (MIP) of magnetic resonance angiography (MRA) in magnetic resonance imaging (MRI). The system 10 is used to provide a simulated MIP image from a T1-weighted image input without the use of a contrast agent. The solutions of the disclosure have a wide range of applications in hospitals, radiology centers, and research institutions. These applications include, but are not limited to:

[0101] (1) Disease diagnosis: non-invasive imaging of intracranial blood vessels;

[0102] (2) Rapid treatment monitoring: monitoring changes in the cerebral vasculature of patients after treatment for cerebrovascular diseases;

[0103] (3) Technical discovery: Scientists and research teams can use this technology to discover new treatment paradigms and better understand potential cerebrovascular mechanisms.

[0104] Some advantages of the embodiments of the disclosure over the prior art include:

[0105] (1) Compared with contrast-enhanced MRA-MRI systems, it does not involve contrast agents.

[0106] (2) Compared with non-contrast MRI-MRA systems, the disclosure provides a method for quickly generating MIP images (completed within a few seconds) and does not need to worry about the impact of patient movement on the generated images.

[0107] (3) Compared with the existing techniques of generating MRA from single or multiple non-contrast images using deep learning, it does not involve the use of advanced MIP generation software, which sometimes provides disjoint vasculature due to changes in image contrast and brightness.

[0108] (4) Publicly generate cerebral vascular tree images directly.

[0109] (5) It provides an inexpensive method for diagnosing cerebrovascular diseases. vi. The technology is easily available and can be integrated into any portable device for easy access.

[0110] Embodiments of the present disclosure can provide a novel and creative technology that combines image processing technology, NIFTI format (Neuroimaging Information Technology Initiative), deep learning and artificial intelligence to generate maximum intensity projection (MIP) images for magnetic resonance angiography (MRA) images by processing selected inputs obtained without the use of contrast agents, thereby diagnosing or monitoring treatment of various cerebrovascular diseases without the use of harmful contrast agents.

[0111] refer to Figure 12 , Figure 12 is a block diagram illustrating workflow steps and components of a method 200 according to another embodiment of the present disclosure. In this embodiment, the method 200 includes the following steps:

[0112] (1) In step 202 , a single non-contrast (SNC) image 204 and a corresponding MRA image 206 are collected from a database 208 .

[0113] (2) In the pre-processing module, the acquired images are pre-processed by visual inspection 212 , voxel normalization 210 , and template registration 214 & 218 .

[0114] (3) The MRA image is post-processed into an MIP image 216 .

[0115] (4) In step 220, the MIP image and the corresponding SNC image are stored in the input module.

[0116] (5) Separate the image pairs into a series of training and testing data 224. The SNC image and the corresponding MIP image 222 are input into the model 226 of the training module.

[0117] (6) In training module 226, model 120 is trained to learn the generation of the simulated MIP images in step 234, using the SNC images as model inputs and the simulated MIP images as target outputs. Adjustable parameters of the model are adjusted so that the output images generated by the model are optimized to resemble the training data targets.

[0118] (7) The simulated MIP image is tested against the corresponding MIP image previously input into the model 120, which is a criterion for validating the model in the validation module 232.

[0119] (8) Then, in step 236, the generated simulated MIP image is stepped through in the database 208.

[0120] Figure 13 A method 250 according to an embodiment of the present disclosure for generating a simulated MIP image from a T1-weighted image. The method 250 includes the following steps:

[0121] (1) In step 252, the SNC image 254 collected from the MRI scanner is transferred to the input module in NIFTI format.

[0122] (2) In step 258, the system 199 preprocesses the SNC image through template registration 257 and voxel normalization 256.

[0123] (3) The SNC image is transferred to the trained model 120, and the trained model 120 receives the SNC image as input in step 260 and provides the MIP image 262 as output.

[0124] (4) In step 264, the generated data is transferred to the medical archive or stored in the output storage module.

[0125] In another embodiment of the present disclosure, a system 100 is provided for performing a method for generating a simulated MIP image of a patient's brain based on a single non-contrast (SNC) MR image of the brain without injecting a contrast agent into the brain. The system includes the following modules:

[0126] (1) An input image module for storing the SNC image of the patient's brain.

[0127] (2) A preprocessing module for receiving the SNC image, preparing and generating the criteria required by the model to effectively extract and understand its features.

[0128] (3) A simulated MIP generation model compartment for receiving the patient's SNC image and for generating a pseudo MIP image corresponding to each SNC image.

[0129] (4) A deep learning platform for operating and constructing the connected model.

[0130] (5) A training module for receiving training data and transferring the training data to the deep learning platform, whereby the adjustable parameters of the model can be adjusted to optimize the simulated MIP image.

[0131] (6) A testing module, which is used to communicate with the training module and the deep learning platform to receive test data, thereby validating the pre-trained performance criteria for the simulated MIP images generated by the model.

[0132] (7) An output storage compartment, which is used to receive the simulated MIP images generated by the model.

[0133] Although not required, the embodiments described with reference to the accompanying drawings can be implemented as an application programming interface (API) or a series of libraries used by developers, or can be included in another software application, such as an operating system of a terminal or a personal computer or an operating system of a portable computing device. Generally, since program modules include routines, programs, objects, components, and data files that assist in performing specific functions, those skilled in the art will understand that the functions of a software application can be distributed among multiple routines, objects, or components to achieve the same functions required herein.

[0134] It should also be understood that in cases where the methods and systems of the present disclosure are implemented entirely or partially by a computing system, any suitable computing system architecture can be used. This will include edge computing devices, stand-alone computers, network computers, cloud-based computing devices, and dedicated hardware devices. When using the terms "computing system" and "computing device", these terms are intended to cover any suitable configuration of computer hardware capable of implementing the described functions.

[0135] Those skilled in the art will understand that various changes and / or modifications can be made to the present disclosure as shown in the specific embodiments without departing from the spirit or scope of the present disclosure as broadly described. Therefore, this embodiment is considered illustrative in all respects and not restrictive.

[0136] Unless otherwise specified, any reference to prior art contained herein should not be construed as an admission that such information is common general knowledge.

Claims

1. A method for synthesizing an MRA image and its MIP image MIP using a single acquired contrast MR image (T1-weighted MR image) by a system, characterized in that, The system has at least a processor and a memory therein to execute instructions of an artificial intelligence engine, and the artificial intelligence engine is configured as a UNet model stored in the memory of the system; the UNet model includes: An encoder having a plurality of layer blocks, each layer block of the encoder including one or more convolutional layers, each of the convolutional layers being associated with an activation layer and a downsampling layer; A decoder having a plurality of layer blocks, each layer block of the decoder including an upsampling layer and one or more convolutional layers, each of the convolutional layers being associated with an activation layer; Skip connections for associating one of the layer blocks of the encoder with one of the layer blocks of the decoder at corresponding multi-scale resolution levels; Wherein, the encoder is configured to extract features from the T1-weighted MR image for the decoder to combine the output from the encoder and the extracted image features at multi-scale resolution levels through the skip connections to generate an MRA image and its MIP image.

2. The method according to claim 1, wherein Wherein, The decoder includes an output layer to generate an image having the same resolution as the input image.

3. The method according to claim 2, wherein Wherein, The output layer includes a single output convolutional layer followed by an output activation layer.

4. The method according to claim 3, wherein Wherein, The single output convolutional layer is a 1×1 convolutional layer with a stride of 1.

5. The method according to claim 4, wherein Wherein, The output activation layer is configured to perform a hyperbolic tangent (tanh) operation.

6. The method according to claim 1, characterized in that, Wherein, The encoder and the decoder are configured to perform a cross-sequence conversion from a T1-weighted image to an MRA or MIP image composed of 19 convolutional layers.

7. The method according to claim 1, wherein Wherein, The encoder is configured to receive an image including three dimensions and one or more color channels.

8. The method according to claim 1, characterized in that, Wherein, One or more layer blocks of the encoder include two 3x3 convolutional layers repeated five times, with a stride of 2 voxels.

9. The method according to claim 1, characterized in that, Wherein, The layer block of the encoder immediately before the decoder includes a single convolutional layer.

10. The method according to claim 1, characterized in that, Wherein, Zero-padding technique is implemented before each convolutional layer.

11. The method according to claim 1, wherein Wherein, The activation layer is configured to perform a linear rectification function through one or more rectified linear units (ReLU).

12. The method according to claim 1, wherein Wherein, The downsampling includes a 2×2×2 max pooling operation with a stride of 2 voxels.

13. The method according to claim 1, wherein Wherein, Each of the convolutional layers is configured to process input data using a plurality of convolutional filters.

14. The method according to claim 1, characterized in that, Wherein, In the encoder, from the first layer block to the last layer block, the number of convolutional filters doubles.

15. The method according to claim 1, characterized in that, Wherein, The upsampling layer of the decoder is configured to perform nearest neighbor interpolation to increase the image size through each layer block within the decoder.

16. The method according to claim 1, wherein Wherein, One or more convolutional layers of the decoder use random initialization and unequal kernel sizes.

17. The method according to claim 1, wherein Wherein, The skip connections are configured to copy and concatenate features generated from one of the layer blocks of the encoder to one of the layer blocks of the decoder at corresponding multi-scale resolution levels.

18. The method according to claim 1, wherein Wherein, The training batch size used by the UNet model is 4.

19. The method according to claim 1, characterized in that, Wherein, The UNet model is trained with consecutive MRA images in a different way from the input image and is trained for more than 100 epochs.

20. A system for converting a T1-weighted image into a vascular system image of the brain based on deep learning, characterized in that, Including: A memory for storing instructions and a processor for executing the instructions stored in the memory; The processor executes an artificial intelligence engine that is configured to store a UNet model in the memory of the system; wherein, the UNet model includes: An encoder having a plurality of layer blocks, each layer block of the encoder including one or more convolutional layers, each of the convolutional layers being associated with an activation layer and a downsampling layer; A decoder having a plurality of layer blocks, each layer block of the decoder including an upsampling layer and one or more convolutional layers, each of the convolutional layers being associated with an activation layer; Skip connections for associating one of the layer blocks of the encoder with one of the layer blocks of the decoder at corresponding multi-scale resolution levels; Wherein, the encoder is configured to extract features from the T1-weighted MR image for the decoder to combine the output from the encoder and the extracted image features at multi-scale resolution levels through the skip connections to generate an MRA image and its MIP image.

Images