A 3D-TOF-MRA image generation model training method and device

Abstract

The application discloses a 3D-TOF-MRA image generation model training method and device, acquires and processes T1w images, T2w images, Flair images and TOF images, splices the first three preprocessed images and the TOF image and inputs the discriminator to obtain a real discrimination image; meanwhile, the first three spliced images are input into a generator of the Unet architecture to obtain a generator output image; the generator output image and the first three images are spliced and input into the discriminator to obtain a generated discrimination image; by comparing the loss of the real discrimination image and the all-1 matrix, the loss of the generated discrimination image and the all-0 matrix, and the loss of the generator output image and the TOF image, the discriminator and the generator weights are updated in reverse propagation until a high-quality 3D-TOF-MRA image is generated. The image generation model trained by the application can generate a 3D-TOF-MRA image with higher quality and more accurate details.

Description

Technical Field

[0001] This application relates to the field of medical image generation technology, specifically to a 3D-TOF-MRA image generation model training method and apparatus. Background Technology

[0002] 3D-TOF-MRA (Three-Dimensional Time-of-Flight Magnetic Resonance Angiography) is a commonly used magnetic resonance imaging technique for vascular imaging. Due to its advantages such as not requiring radioactive contrast agents, high imaging resolution, and multi-angle imaging, it is widely used in neuroscience, cardiovascular medicine, and surgical planning to detect vascular stenosis, abnormalities, and hemodynamic changes. However, because 3D-TOF-MRA scans are relatively long, patient movement can cause image blurring or artifacts, affecting image quality. Furthermore, since this technique relies on the TOF (Time-of-Flight) effect for imaging, the choice of scanning angle also affects the quality and accuracy of the final image.

[0003] With the advancement of deep learning technology, many classic Convolutional Neural Network (CNN) architectures (such as U-Net or DenseNet) have been used in medical image processing, such as segmenting lesions and removing image noise. However, for 3D-TOF-MRA images, existing technologies mainly focus on improving low-quality, low-scan-time 3D-TOF-MRA images. For example, Wicaksono KP, Fujimoto K, Fushimi Y, et al. Super-resolution application of generative adversarial network on brain time-of-flight MRangiography: image quality and diagnostic utility evaluation[J]. European Radiology, 2023, 33(2): 936-946. discloses a method for super-resolution reconstruction using generative adversarial networks in brain time-of-flight magnetic resonance angiography. However, this method is based on low-quality images and cannot fundamentally avoid motion-induced artifacts and the influence of scanning angle on image quality. You SH, Cho Y, Kim B, et al. Deep Learning–Based Synthetic TOF-MRA Generation Using Time-Resolved MRA in Fast Stroke Imaging[J]. American Journal of The paper *Neuroradiology*, 2023, 44(12):1391-1398, discloses a method for generating synthetic TOF-MRA using time-resolved MRA in rapid stroke imaging based on deep learning. This method uses a CycleGAN (Cycle Generative Adversarial Network) network structure to perform style transfer on 4D-TWIST-MRA images to generate MIP (Maximum Intensity Projection) images of TOF-MRA. However, this method only generates cross-sectional MIP images, which is detrimental to the detection of diseases such as vascular stenosis and aneurysms. Therefore, it is necessary to develop an efficient, accurate, and high-resolution 3D-TOF-MRA generation method. Summary of the Invention

[0004] To address this issue, this application provides a 3D-TOF-MRA image generation model training method and apparatus to solve the problems of inaccurate and poor-quality 3D-TOF-MRA images generated by existing technologies.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] Firstly, a method for training a 3D-TOF-MRA image generation model includes:

[0007] Step 1: Acquire T1-weighted images, T2-weighted images, Flair images, and TOF-MRA standard images of the same area;

[0008] Step 2: Preprocess the T1-weighted image, the T2-weighted image, the Flair image, and the TOF-MRA standard image;

[0009] Step 3: The preprocessed T1-weighted image, T2-weighted image, and Flair image are stitched together with the TOF-MRA standard image and input into the discriminator of the pre-constructed conditional generative adversarial network to obtain the real discriminant image output by the discriminator;

[0010] Step 4: The preprocessed T1-weighted image, T2-weighted image, and Flair image are stitched together and input into the generator of the pre-constructed conditional generative adversarial network to obtain the generator output image; wherein, the generator is a Unet architecture network;

[0011] Step 5: The preprocessed T1-weighted image, T2-weighted image, and Flair image are stitched together with the generator output image and input into the discriminator to obtain the generated discriminant image output by the discriminator;

[0012] Step 6: Calculate the loss function of the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and backpropagate to update the discriminator weights;

[0013] Step 7: Calculate the loss function between the generator output image and the TOF-MRA standard image, and the loss function between the generated discriminant image and the all-1 matrix, and backpropagate to update the generator weights;

[0014] Step 8: Repeat steps 6 and 7 until a model capable of generating high-quality 3D-TOF-MRA images is trained.

[0015] Preferably, step 2 specifically includes: registering the T1-weighted image, the T2-weighted image, and the Flair image according to the TOF-MRA standard image, and normalizing the T1-weighted image, the T2-weighted image, the Flair image, and the TOF-MRA standard image.

[0016] Preferably, in step 3, the discriminator adopts a multi-layer convolutional block stacking structure.

[0017] Preferably, in step 3, the discriminator uses mean square error loss.

[0018] Preferably, in step 4, the Unet architecture network is Unet++, AttentionU-Net, or Mamba-UNet.

[0019] Preferably, in step 4, the basic module of the generator is VGG_Block, which includes instance normalization, convolutional spectrum normalization, and dropout layer.

[0020] Preferably, the activation function of the VGG_Block is LeakReLU.

[0021] Preferably, in step 4, the generator employs a hybrid loss of L1 loss, structural similarity loss, and maximum projection loss.

[0022] Preferably, the maximum projection mixing loss is the maximum projection mixing loss based on Gumbel random sampling.

[0023] Secondly, a 3D-TOF-MRA image generation model training device includes:

[0024] The data acquisition module is used to acquire T1-weighted images, T2-weighted images, Flair images, and TOF-MRA standard images of the same body part.

[0025] The data preprocessing module is used to preprocess the T1-weighted image, the T2-weighted image, the Flair image, and the TOF-MRA standard image;

[0026] The generator module is used to stitch together the preprocessed T1-weighted image, the T2-weighted image, and the Flair image, and input them into the generator of a pre-constructed conditional generative adversarial network to obtain the generator output image; wherein, the generator is a Unet architecture network;

[0027] The discriminator module is used to stitch the preprocessed T1-weighted image, the T2-weighted image, and the Flair image with the TOF-MRA standard image, and input the result into the discriminator of the pre-constructed conditional generative adversarial network to obtain the real discriminant image output by the discriminator.

[0028] The preprocessed T1-weighted image, T2-weighted image, and Flair image are then stitched together with the generator output image and input into the discriminator to obtain the generated discriminant image output by the discriminator.

[0029] The model training module is used to calculate the loss function of the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and to backpropagate to update the discriminator weights;

[0030] And calculate the loss function between the generator output image and the TOF-MRA standard image, and the loss function between the generated discriminant image and the all-1 matrix, and backpropagate to update the generator weights;

[0031] And repeatedly update the discriminator weights and the generator weights until a high-quality 3D-TOF-MRA image generation model is trained.

[0032] Compared with the prior art, this application has at least the following beneficial effects:

[0033] This application provides a method and apparatus for training a 3D-TOF-MRA image generation model. The method involves acquiring and preprocessing T1-weighted, T2-weighted, and FLAIR images, as well as a standard TOF-MRA image. The preprocessed first three images are then concatenated with the TOF-MRA image and input into a discriminator to obtain a true discriminant image. Simultaneously, the concatenated first three images are input into a generator based on a Unet architecture to obtain the generator's output image. This generator's output image is then concatenated with the first three images and input into the discriminator to obtain a generated discriminant image. By comparing the losses of the true discriminant image with an all-1 matrix, the generated discriminant image with an all-0 matrix, and the generator's output image with the TOF-MRA image, backpropagation is used to update the weights of the discriminator and generator. This process is repeated until a high-quality 3D-TOF-MRA image is generated. The image generation model trained using this application can generate higher-quality, more detailed 3D-TOF-MRA images, which is of great significance for accurate lesion localization and diagnosis in clinical practice. Attached Figure Description

[0034] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0035] Figure 1 This is a flowchart of a 3D-TOF-MRA image generation model training method provided in Embodiment 1 of this application;

[0036] Figure 2 This is a schematic diagram of the data preprocessing structure provided in Embodiment 1 of this application;

[0037] Figure 3 This is a schematic diagram of the Unet++ network structure provided in Embodiment 1 of this application;

[0038] Figure 4 This is a schematic diagram of the VGG_Block structure provided in Embodiment 1 of this application;

[0039] Figure 5 This is a schematic diagram of the training process structure provided in Embodiment 1 of this application;

[0040] Figure 6 This is a schematic diagram of a 3D-TOF-MRA image generation model training device provided in Embodiment 2 of this application. Detailed Implementation

[0041] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "comprising," "including," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0043] The terms used in this application, such as "upper," "lower," "left," "right," and "middle," are generally used to indicate the general relative positional relationship for the purpose of intuitive understanding by referring to the accompanying drawings, and are not absolute limitations on the positional relationship in the actual product.

[0044] Example 1

[0045] Please see Figure 1This embodiment provides a training method for a 3D-TOF-MRA image generation model, including:

[0046] S1: Acquire T1-weighted images, T2-weighted images, Flair images, and TOF-MRA standard images of the same body region;

[0047] Specifically, this involves acquiring MRI (magnetic resonance imaging) data of the same location in different modalities. These different modalities include T1-weighted images (abbreviated as: I...). T1-w Or T1w), T2-weighted images (abbreviated as: I T2-w Or T2w), Flair image (abbreviated as: I falir ) and TOF-MRA standard images (abbreviated as: I TOF Or TOF).

[0048] S2: Preprocess the T1-weighted image, T2-weighted image, Flair image, and TOF-MRA standard image;

[0049] Specifically, due to differences in patient position and machine scanning parameters during image data acquisition, the positions of images from different modalities do not completely correspond, thus requiring image registration. This embodiment first uses I... TOF For template I T1-w I T2-w and I falir Registration was performed using nearest neighbor interpolation, and then, for I... T1-w I T2-w I falir and I TOF Normalization is performed to adjust the data range to between 0 and 1, such as... Figure 2 As shown.

[0050] S3: The preprocessed T1-weighted image, T2-weighted image, and Flair image are stitched together with the TOF-MRA standard image and input into the discriminator of the pre-built conditional generative adversarial network to obtain the real discriminant image output by the discriminator;

[0051] Specifically, the pre-built conditional generative adversarial network (cGAN) in this example consists of two parts: a generator G and a discriminator D. The discriminator of the pre-built conditional generative adversarial network adopts a multi-layer convolutional block stacking structure.

[0052] This step will I T1-w I T2-wI falir With and I TOF The concatenation is used as the input F of the discriminator D. i This can be expressed as a formula:

[0053] F real =cat(I T1-w I T2-w I fali r,I TOF ).

[0054] S4: The preprocessed T1-weighted image, T2-weighted image, and Flair image are concatenated and input into the generator of the pre-built conditional generative adversarial network to obtain the generator output image; wherein, the generator is a Unet architecture network;

[0055] Specifically, the generator G's underlying architecture is a Unet architecture network, such as Unet++, Attention U-Net, or Mamba-UNet. In this embodiment, the preferred generator architecture is Unet++, such as... Figure 3 As shown, Unet++ introduces a multi-level feature fusion mechanism on top of U-Net. Traditional U-Net only has a single level of feature fusion, while Unet++ connects feature maps from different levels, enabling the network to utilize both shallow and deep features simultaneously, thereby improving the expressive power of features and the ability to acquire semantic information.

[0056] It should be noted that for the background architecture Unet++ used in cGAN, the number of downsampling times can be increased or decreased, or corresponding deep supervision can be applied to the output of the intermediate layers.

[0057] In this embodiment, the basic module in generator G is changed to VGG Block. To improve generalization, VGG Block introduces instance normalization (InstanceNorm), convolutional spectral normalization (Conv), and dropout layers. The activation function uses LeakReLU to improve model training speed. The specific structure is as follows: Figure 4 As shown.

[0058] Please see Figure 5 This step will generate images I from the three modalities. T1-w I T2-w and I falir The concatenation of these two parts forms the input I of the generator. input This can be expressed as a formula:

[0059] I input =cat(I T1-w I T2-w Ifalir ),

[0060] The resulting generator output image is G(I) input ).

[0061] It should be noted that when concatenating the three input modal data, the input methods and fusion strategies of the three can be changed. For example, attention mechanisms can be used to assist fusion, or multiple encoders can be used to post-process the output features.

[0062] S5: The preprocessed T1-weighted image, T2-weighted image, and Flair image are concatenated with the generator output image and input into the discriminator to obtain the generated discriminant image output by the discriminator;

[0063] For details, please refer to [link / reference]. Figure 5 This step will I T1-w I T2-w I falir With the output G(I) of generator G input The concatenation is used as the input F of the discriminator D. i This can be expressed as a formula:

[0064] F fake =cat(I T1-w， I T2-w I falir ,G(I input )).

[0065] S6: Calculate the loss function between the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and backpropagate to update the discriminator weights;

[0066] In this embodiment, the loss function of the discriminator D is the Mean Square Error (MSE) loss. For the discriminator D, in the initial stage of a training cycle, the discriminator D needs to distinguish between real and generated images; therefore, the loss function is:

[0067] f D_loss1 =f L2 (D(I TOF ), true)+f L2 (D(G(I input )), false)

[0068] Where true and false are matrices of all 1s and all 0s of the same size as the output of the discriminator D, respectively.

[0069] S7: Calculate the loss function between the generator output image and the TOF-MRA standard image, as well as the loss function between the generated discriminant image and the all-1 matrix, and backpropagate to update the generator weights;

[0070] Specifically, the input image of generator G is I. input The output image is G(I) input To obtain a better image, the output of the generator G is also constrained, and the output image G(I) is constrained. input ) and standard image I TOF Calculate the loss and backpropagate to update the weights of generator G.

[0071] In this embodiment, the generator G uses a hybrid loss function consisting of L1 loss, Structural Similarity Index (SSIM) loss, and Maximum Projection (MIP) loss. The L1 loss and SSIM loss are expressed as follows:

[0072]

[0073]

[0074] Among them, f ssim and f L1 Let G(I) represent the SSIM loss and L1 loss respectively, and let a represent the output image, i.e., G(I) input b represents the true standard image, i.e., I TOF μ a μ b σ a and σ b Let a and b represent the mean and standard deviation, respectively; c1 and c2 are two constants; and n represents the number of pixels in a and b. i b i This represents the value of the i-th pixel in a and b.

[0075] For MIP loss, first calculate I TOF Maximum value image I max With the generator output image G(I) input Maximum value image

[0076] I max =max(I TOF )

[0077] Rand Gumbel =-ln(-ln(U+1) -20 )+1 -20 )

[0078]

[0079] Where U ~ U(0,1), T is the temperature coefficient, and Gumbel random sampling is used here in order to enable the gradient to propagate in the back.

[0080] Then calculate the MIP loss f MIP :

[0081]

[0082] During the adversarial phase, the output of generator G needs to be closer to I. TOF The loss function at this point is:

[0083] f G_loss2 =f L2 (D(G(I input )), true).

[0084] Therefore, the loss function of the final generator G is expressed as:

[0085] f G_loss =f L1 +f SSIM +f MIP +f G_loss2 .

[0086] It should be noted that other loss functions can also be used in this step, such as any combination of mean squared error loss, L1 loss, PSNR loss, and SSIM loss. Furthermore, other regularization or loss terms can be introduced, such as contrastive loss and perceptual loss, to further improve denoising performance and image quality. Additionally, the Gumbel sampling in the MIP loss can be modified to use other sampling methods that are sensitive to the distribution of maximum values.

[0087] This embodiment provides a 3D-TOF-MRA image generation model training method that uses three readily available MRI images—T1-weighted, T2-weighted, and Flair—to train a model capable of generating high-quality 3D-TOF-MRA images. It leverages the adversarial nature of conditional generative adversarial networks (GANs) to enhance the generalization properties of the generative model, enabling it to perform well across a wider range of data. The preferred background network architecture for the generator is Unet++, which introduces multi-level path connections and dense skip connections. These connection mechanisms improve the efficiency of information propagation and path selection within the network, facilitating better integration and utilization of multi-scale features. Furthermore, considering the common use of MIP images for vascular observation in diagnosis, this embodiment introduces a Gumbel-based random sampling MIP loss to improve the quality of the generated 3D-TOF-MRA images. In 3D-TOF-MRA images, blood vessels exhibit high signal intensity. Gumbel sampling can approximate the distribution of high signal intensity in the image, thereby strengthening the constraint on high signal intensity areas, which are clinically significant parts. Therefore, while generating the basic structure, it can increase the model's attention to the details of blood vessels in 3D-TOF-MRA images, thereby improving the quality of the generated images and the accuracy of blood vessel measurements.

[0088] Compared to existing technologies that enhance low-dose 3D-TOF-MRA images or generate 3D-TOF-MRA images using images from a single modality, this embodiment utilizes three modalities with different characteristics to generate higher-quality 3D-TOF-MRA images with more precise details. This is of great significance for the accurate localization and diagnosis of lesions in clinical practice.

[0089] Example 2

[0090] Please see Figure 6 This embodiment provides a 3D-TOF-MRA image generation model training device, including:

[0091] The data acquisition module is used to acquire T1-weighted images, T2-weighted images, Flair images, and TOF-MRA standard images of the same body part.

[0092] The data preprocessing module is used to preprocess the T1-weighted image, the T2-weighted image, the Flair image, and the TOF-MRA standard image;

[0093] The generator module is used to stitch together the preprocessed T1-weighted image, the T2-weighted image, and the Flair image, and input them into the generator of a pre-constructed conditional generative adversarial network to obtain the generator output image; wherein, the generator is a Unet architecture network;

[0094] The discriminator module is used to stitch the preprocessed T1-weighted image, the T2-weighted image, and the Flair image with the TOF-MRA standard image, and input the result into the discriminator of the pre-constructed conditional generative adversarial network to obtain the real discriminant image output by the discriminator.

[0095] The preprocessed T1-weighted image, T2-weighted image, and Flair image are then stitched together with the generator output image and input into the discriminator to obtain the generated discriminant image output by the discriminator.

[0096] The model training module is used to calculate the loss function of the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and to backpropagate to update the discriminator weights;

[0097] And calculate the loss function between the generator output image and the TOF-MRA standard image, and the loss function between the generated discriminant image and the all-1 matrix, and backpropagate to update the generator weights;

[0098] And repeatedly update the discriminator weights and the generator weights until a high-quality 3D-TOF-MRA image generation model is trained.

[0099] For details on the specific implementation of each module in a 3D-TOF-MRA image generation model training device, please refer to the above description of the limitations of a 3D-TOF-MRA image generation model training method, which will not be repeated here.

[0100] The technical features of the above embodiments can be combined in any way (as long as there is no contradiction in the combination of these technical features). For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; these embodiments not explicitly written should also be considered to be within the scope of this specification.

Claims

1. A training method for a 3D-TOF-MRA image generation model, characterized in that, include: Step 1: Acquire T1-weighted images, T2-weighted images, Flair images, and TOF-MRA standard images of the same area; Step 2: Preprocess the T1-weighted image, the T2-weighted image, the Flair image, and the TOF-MRA standard image; Step 3: The preprocessed T1-weighted image, T2-weighted image, and Flair image are stitched together with the TOF-MRA standard image and input into the discriminator of the pre-constructed conditional generative adversarial network to obtain the real discriminant image output by the discriminator; Step 4: The preprocessed T1-weighted image, T2-weighted image, and Flair image are stitched together and input into the generator of the pre-constructed conditional generative adversarial network to obtain the generator output image; wherein, the generator is a Unet architecture network; Step 5: The preprocessed T1-weighted image, T2-weighted image, and Flair image are stitched together with the generator output image and input into the discriminator to obtain the generated discriminant image output by the discriminator; Step 6: Calculate the loss function of the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and backpropagate to update the discriminator weights; Step 7: Calculate the loss function between the generator output image and the TOF-MRA standard image, and the loss function between the generated discriminant image and the all-1 matrix, and backpropagate to update the generator weights; Step 8: Repeat steps 6 and 7 until a model capable of generating high-quality 3D-TOF-MRA images is trained.

2. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that, Step 2 specifically includes: registering the T1-weighted image, the T2-weighted image, and the Flair image according to the TOF-MRA standard image, and normalizing the T1-weighted image, the T2-weighted image, the Flair image, and the TOF-MRA standard image.

3. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that, In step 3, the discriminator adopts a multi-layer convolutional block stacking structure.

4. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that, In step 3, the discriminator uses mean square error loss.

5. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that, In step 4, the Unet architecture network is Unet++, Attention U-Net, or Mamba-UNet.

6. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that, In step 4, the basic module of the generator is VGG_Block, which includes instance normalization, convolutional spectrum normalization, and dropout layer.

7. The 3D-TOF-MRA image generation model training method according to claim 6, characterized in that, The activation function for VGG_Block is LeakReLU.

8. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that, In step 4, the generator employs a hybrid loss of L1 loss, structural similarity loss, and maximum projection loss.

9. The 3D-TOF-MRA image generation model training method according to claim 8, characterized in that, The maximum projection mixing loss is the maximum projection mixing loss based on Gumbel random sampling.

10. A 3D-TOF-MRA image generation model training device, characterized in that, include: The data acquisition module is used to acquire T1-weighted images, T2-weighted images, Flair images, and TOF-MRA standard images of the same body part. The data preprocessing module is used to preprocess the T1-weighted image, the T2-weighted image, the Flair image, and the TOF-MRA standard image; The generator module is used to stitch together the preprocessed T1-weighted image, the T2-weighted image, and the Flair image, and input them into the generator of a pre-constructed conditional generative adversarial network to obtain the generator output image; wherein, the generator is a Unet architecture network; The discriminator module is used to stitch the preprocessed T1-weighted image, the T2-weighted image, and the Flair image with the TOF-MRA standard image, and input the result into the discriminator of the pre-constructed conditional generative adversarial network to obtain the real discriminant image output by the discriminator. The preprocessed T1-weighted image, T2-weighted image, and Flair image are then stitched together with the generator output image and input into the discriminator to obtain the generated discriminant image output by the discriminator. The model training module is used to calculate the loss function of the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and to backpropagate to update the discriminator weights; And calculate the loss function between the generator output image and the TOF-MRA standard image, and the loss function between the generated discriminant image and the all-1 matrix, and backpropagate to update the generator weights; And repeatedly update the discriminator weights and the generator weights until a high-quality 3D-TOF-MRA image generation model is trained.

Images