3D-TOF-MRA image generation model training method and device

Abstract

The invention discloses a 3D-TOF-MRA image generation model training method and device, and the method comprises the steps: obtaining and processing a T1w image, a T2w image, a Flair image and a TOF image, splicing the first three images after preprocessing with the TOF image, and inputting the spliced images into a discriminator to obtain a real discrimination image; meanwhile, splicing the first three images and then inputting the spliced images into a generator of the Unet architecture to obtain a generator output image; splicing the output image of the generator with the first three images, and inputting the spliced images into a discriminator to obtain a generated discrimination image; and the weight of the discriminator and the weight of the generator are updated through back propagation by comparing the real discriminating image with the all-1 matrix, generating the loss of the discriminating image and the all-0 matrix and the loss of the output image of the generator and the TOF image until the high-quality 3D-TOF-MRA image is generated. The 3D-TOF-MRA image with higher quality and more accurate details can be generated by using the image generation model trained by the method.

Description

Technical Field

[0001] The present application relates to the technical field of medical image generation, and in particular to a 3D-TOF-MRA image generation model training method and device. Background Art

[0002] 3D-TOF-MRA (Three-Dimensional Time-of-Flight Magnetic Resonance Angiography) is a magnetic resonance imaging technology commonly used for vascular imaging. It is widely used in neuroscience, cardiology, surgical planning and other fields to detect vascular stenosis, abnormalities and hemodynamic changes because it does not require the use of radioactive contrast agents, has high imaging resolution and multi-angle imaging. However, since the scanning time of 3D-TOF-MRA is relatively long, the patient's movement may cause image blur or artifacts, affecting the imaging quality; and this technology relies on the TOF (Time-of-Flight) effect for imaging, so the choice of scanning angle will also affect the quality and accuracy of the final image.

[0003] With the advancement of deep learning technology, many classic convolutional neural network (CNN) architectures (such as U-Net or DenseNet) have been used in medical image processing, such as lesion segmentation and image noise removal. However, for 3D-TOF-MRA images, the existing technologies mainly focus on improving low-quality and low-scanning-time 3D-TOF-MRA images. For example, Wicaksono KP, Fujimoto K, Fushimi Y, et al. Super-resolution application of generative adversarial network on brain time-of-flight MRangiography: image quality and diagnostic utility evaluation [J]. European Radiology, 2023, 33 (2): 936-946. discloses a method for super-resolution reconstruction using a generative adversarial network in brain time-of-flight magnetic resonance angiography. However, this method is based on low-quality images and cannot fundamentally avoid the problem of motion-induced artifacts and scanning angles affecting imaging quality; You SH, Cho Y, Kim B, et al. Deep Learning–Based Synthetic TOF-MRA Generation Using Time-Resolved MRA in Fast Stroke Imaging [J]. American Journal of Neuroradiology, 2023, 44(12): 1391-1398. discloses a method for generating synthetic TOF-MRA using time-resolved MRA in rapid stroke imaging based on deep learning. The method uses the CycleGAN (Cycle Generative Adversarial Network) network structure to perform style conversion on 4D-TWIST-MRA images to generate MIP (Maximum intensity projection) images of TOF-MRA. However, this method only generates cross-sectional MIP images, which is not conducive to the detection of diseases such as vascular stenosis and aneurysm. Therefore, it is necessary to develop an efficient, accurate, and high-resolution 3D-TOF-MRA generation method. Summary of the invention

[0004] To this end, the present application provides a 3D-TOF-MRA image generation model training method and device to solve the problem of inaccurate and poor quality 3D-TOF-MRA images generated by the prior art.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, a 3D-TOF-MRA image generation model training method includes:

[0007] Step 1: Obtain T1-weighted images, T2-weighted images, Flair images, and TOF-MRA standard images of the same part;

[0008] Step 2: preprocessing the T1-weighted image, the T2-weighted image, the Flair image and the TOF-MRA standard image;

[0009] Step 3: splicing the preprocessed T1-weighted image, the T2-weighted image and the Flair image with the TOF-MRA standard image, and inputting them into the discriminator of the pre-built conditional generative adversarial network to obtain the real discriminant image output by the discriminator;

[0010] Step 4: splicing the preprocessed T1-weighted image, the T2-weighted image and the Flair image, and inputting them into a generator of a pre-built conditional generative adversarial network to obtain a generator output image; wherein the generator is a Unet architecture network;

[0011] Step 5: splicing the preprocessed T1-weighted image, the T2-weighted image and the Flair image with the generator output image, and inputting them into the discriminator to obtain a generated discriminant image output by the discriminator;

[0012] Step 6: Calculate the loss function between the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and back-propagate to update the discriminator weights;

[0013] Step 7: Calculate the loss function between the generator output image and the TOF-MRA standard image and the loss function between the generated discriminant image and the all-1 matrix, and back-propagate to update the generator weights;

[0014] Step 8: Repeat steps 6 and 7 until a model capable of generating high-quality 3D-TOF-MRA images is trained.

[0015] Preferably, step 2 specifically includes: registering the T1-weighted image, the T2-weighted image and the Flair image according to the TOF-MRA standard image, and normalizing the T1-weighted image, the T2-weighted image, the Flair image and the TOF-MRA standard image.

[0016] Preferably, in step 3, the discriminator adopts a multi-layer convolution block stacking structure.

[0017] Preferably, in step 3, the discriminator adopts mean square error loss.

[0018] Preferably, in step 4, the Unet architecture network is Unet++, AttentionU-Net or Mamba-UNet.

[0019] Preferably, in step 4, the basic module of the generator is VGG_Block, and the VGG_Block includes instance normalization, convolutional spectral normalization and a discard layer.

[0020] Preferably, the activation function of the VGG_Block adopts LeakReLU.

[0021] Preferably, in step 4, the generator adopts L1 loss, structural similarity loss and maximum projection mixed loss.

[0022] Preferably, the maximum projection mixed loss is a maximum projection mixed loss based on Gumbel random sampling.

[0023] In a second aspect, a 3D-TOF-MRA image generation model training device includes:

[0024] A data acquisition module is used to acquire T1-weighted images, T2-weighted images, Flair images and TOF-MRA standard images of the same part;

[0025] A data preprocessing module, used for preprocessing the T1-weighted image, the T2-weighted image, the Flair image and the TOF-MRA standard image;

[0026] A generator module, used for splicing the pre-processed T1-weighted image, the T2-weighted image and the Flair image, and inputting them into a generator of a pre-built conditional generative adversarial network to obtain a generator output image; wherein the generator is a Unet architecture network;

[0027] A discriminator module, used to splice the pre-processed T1-weighted image, the T2-weighted image and the Flair image with the TOF-MRA standard image, and input them into the discriminator of the pre-built conditional generative adversarial network to obtain a real discriminant image output by the discriminator;

[0028] and concatenating the preprocessed T1-weighted image, the T2-weighted image and the Flair image with the generator output image, and inputting the concatenated images into the discriminator to obtain a generated discriminant image output by the discriminator;

[0029] A model training module, used to calculate the loss function between the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and back-propagate to update the discriminator weights;

[0030] And calculate the loss function between the generator output image and the TOF-MRA standard image and the loss function between the generated discriminant image and the all-1 matrix, and back-propagate to update the generator weights;

[0031] And repeatedly updating the discriminator weight and the generator weight until a model capable of generating high-quality 3D-TOF-MRA images is trained.

[0032] Compared with the prior art, this application has at least the following beneficial effects:

[0033] The present application provides a 3D-TOF-MRA image generation model training method and device, by acquiring and preprocessing T1-weighted, T2-weighted, Flair images and TOF-MRA standard images; splicing the first three preprocessed images with the TOF-MRA image, inputting the discriminator to obtain the real discriminant image; at the same time, the first three images are spliced ​​and input into the generator of the Unet architecture to obtain the generator output image; the generator output image is spliced ​​with the first three images, and input into the discriminator to obtain the generated discriminant image, by comparing the loss of the real discriminant image with the all-1 matrix, the loss of the generated discriminant image with the all-0 matrix, and the loss of the generator output image with the TOF-MRA image, back propagation updates the discriminator and generator weights, and repeats this process until a high-quality 3D-TOF-MRA image is generated. The image generation model trained by the present application can generate 3D-TOF-MRA images with higher quality and more precise details, which is of great significance for clinical accurate positioning of lesions and diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more intuitively illustrate the prior art and the present application, exemplary drawings are given below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing the present application; for example, those skilled in the art are capable of easily making conventional adjustments or further optimizations to the addition / reduction / attribution division, specific shapes, positional relationships, connection methods, dimensional ratios, etc. of certain units (components) based on the technical concepts and exemplary drawings disclosed in the present application.

[0035] Figure 1 A flow chart of a 3D-TOF-MRA image generation model training method provided in Example 1 of the present application;

[0036] Figure 2 A schematic diagram of the data preprocessing structure provided in Example 1 of the present application;

[0037] Figure 3 A schematic diagram of the Unet++ network structure provided in Example 1 of the present application;

[0038] Figure 4 A schematic diagram of the VGG_Block structure provided in Example 1 of the present application;

[0039] Figure 5 A schematic diagram of the training process structure provided in Example 1 of the present application;

[0040] Figure 6 A schematic diagram of the structure of a 3D-TOF-MRA image generation model training device provided in Example 2 of the present application. DETAILED DESCRIPTION

[0041] The present application is further described below in detail through specific embodiments in conjunction with the accompanying drawings.

[0042] In the description of this application: unless otherwise specified, the meaning of "plurality" is two or more. The terms "first", "second", "third", etc. in this application are intended to distinguish the objects referred to, and do not have special meanings in terms of technical connotations (for example, they should not be understood as emphasizing the importance or order, etc.). Expressions such as "including", "comprising", "having", etc. also mean "not limited to" (certain units, components, materials, steps, etc.).

[0043] The terms such as "upper", "lower", "left", "right", "middle", etc. cited in this application are usually used to indicate the general relative position relationship for the convenience of intuitive understanding by referring to the drawings, and are not absolute limitations on the position relationship in the actual product.

[0044] Embodiment 1

[0045] See also Figure 1, this embodiment provides a 3D-TOF-MRA image generation model training method, including:

[0046] S1: Acquire T1-weighted images, T2-weighted images, Flair images, and TOF-MRA standard images of the same site;

[0047] Specifically, MRI (magnetic resonance imaging) data of different modalities of the same part are obtained, and the MRI data of different modalities include T1-weighted images (abbreviated as: I T1-w or T1w), T2-weighted image (abbreviated as: I T2-w or T2w), Flair image (abbreviated as: I falir ) and TOF-MRA standard image (abbreviated as: I TOF or TOF).

[0048] S2: preprocessing T1-weighted images, T2-weighted images, Flair images and TOF-MRA standard images;

[0049] Specifically, since the patient position and machine scanning parameters are different during the image data acquisition process, the positions of images of different modalities do not completely correspond, so it is necessary to align the images. TOF For template I T1-w ,I T2-w and I falir The nearest neighbor interpolation method is used for registration, and then, I T1-w ,I T2-w ,I falir and I TOF Normalize and adjust the data range to between 0 and 1, such as Figure 2 shown.

[0050] S3: The preprocessed T1-weighted image, T2-weighted image and Flair image are spliced ​​with the TOF-MRA standard image, and input into the discriminator of the pre-built conditional generative adversarial network to obtain the real discriminant image output by the discriminator;

[0051] Specifically, the conditional generative adversarial network (cGAN) pre-constructed in this example includes two parts: a generator G and a discriminator D. The discriminator of the pre-constructed conditional generative adversarial network adopts a multi-layer convolutional block stacking structure.

[0052] This step will T1-w ,I T2-w,I falir With I TOF Concatenate as the input F of the discriminator D i , expressed as:

[0053] F real =cat(I T1-w , I T2-w , I fali r,I TOF ).

[0054] S4: splicing the preprocessed T1-weighted image, T2-weighted image and Flair image, and inputting them into the generator of the pre-built conditional generative adversarial network to obtain the generator output image; wherein the generator is a Unet architecture network;

[0055] Specifically, the basic architecture of the generator G is a Unet architecture network, such as Unet++, Attention U-Net or Mamba-UNet. The preferred generator basic architecture of this embodiment is Unet++, such as Figure 3 As shown in the figure, Unet++ introduces a multi-level feature fusion mechanism based on U-Net. Traditional U-Net only has a single level of feature fusion, while Unet++ connects feature graphs at different levels, allowing the network to use both shallow and deep features at the same time, thereby improving the expressiveness of features and the ability to obtain semantic information.

[0056] It should be noted that for the background architecture Unet++ used by cGAN, the number of downsampling can be increased or decreased, or corresponding deep supervision can be performed on the output of the intermediate layer.

[0057] In this embodiment, the basic module in the generator G is changed to VGG Block. In order to improve generalization, instance normalization (InstanceNorm), convolution (Conv) spectral normalization and dropout layer (dropout) are introduced in VGG_Block, and the activation function uses LeakReLU to increase the model training speed. The specific structure is as follows: Figure 4 shown.

[0058] See also Figure 5 In this step, the images of the three modalities I T1-w ,I T2-w and I falir Spliced ​​together is the input I of the generator input , expressed as:

[0059] I input =cat(I T1-w , I T2-w , Ifalir ),

[0060] The resulting generator output image is G(I input ).

[0061] It should be noted that when splicing the three input modal data, the input method and fusion strategy of the three can be changed, for example: using an attention mechanism to assist fusion or using multiple encoders to post-process the output features.

[0062] S5: concatenate the preprocessed T1-weighted image, T2-weighted image and Flair image with the generator output image, and input them into the discriminator to obtain the generated discriminant image output by the discriminator;

[0063] For details, please refer to Figure 5 In this step, I T1-w ,I T2-w ,I falir With the output G(I input ) is concatenated as the input F of the discriminator D i , expressed as:

[0064] F fake =cat(I T1-w， I T2-w , I falir , G(I input )).

[0065] S6: Calculate the loss function between the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and back-propagate to update the discriminator weights;

[0066] In this embodiment, the loss function of the discriminator D is the mean square error (MSE) loss. For the discriminator D, in a training cycle, in the initial stage, the discriminator D needs to distinguish between the real image and the generated image, so the loss function is:

[0067] f D_loss1 =f L2 (D(I TOF ),true)+f L2 (D(G(I input )),false)

[0068] Among them, true and false are all-1 and all-0 matrices of the same size as the discriminator D output, respectively.

[0069] S7: Calculate the loss function between the generator output image and the TOF-MRA standard image and the loss function between the generated discriminant image and the all-1 matrix, and back-propagate to update the generator weights;

[0070] Specifically, the input image of the generator G is I input , the output image is G(I input ), in order to obtain a better image, the output of the generator G is also constrained, and the output image G(I input ) and standard image I TOF Calculate the loss and backpropagate to update the weights of the generator G.

[0071] The loss function used by the generator G in this embodiment is L1 loss, structural similarity index (SSIM) loss and maximum projection (MIP) mixed loss. Among them, L1 loss and SSIM loss are expressed as follows:

[0072]

[0073]

[0074] Among them, f ssim and f L1 They represent SSIM loss and L1 loss respectively, and a represents the output image, that is, G(I input ), b represents the real standard image, that is, I TOF , μ a , μ b , σ a and σ b Represent the mean and standard deviation of a and b respectively, c1 and c2 are two constants, n represents the number of pixels in a and b, a i 、b i Represents the i-th pixel value in a and b.

[0075] For the MIP loss, first calculate I TOF The maximum value image I max And the generator output image G(I input )'s maximum value image

[0076] I max =max(I TOF )

[0077] Rand Gumbel =-ln(-ln(U+1 -20 )+1 -20 )

[0078]

[0079] Among them, U~U(0,1), T is the temperature coefficient, and Gumbel random sampling is used here to enable the gradient to propagate backward.

[0080] Then calculate the MIP loss f MIP :

[0081]

[0082] In the adversarial stage, the output of the generator G needs to be closer to I TOF , then the loss function is:

[0083] f G_loss2 =f L2 (D(G(I input )),true).

[0084] Therefore, the loss function of the final generator G is expressed as:

[0085] f G_loss =f L1 +f SSIM +f MIP +f G_loss2 .

[0086] It should be noted that this step can also use other loss functions, such as any combination of mean square error loss, L1 loss, PSNR loss, and SSIM loss. In addition, other regularization terms or loss terms, such as contrastive loss and perceptual loss, can also be introduced to further improve the denoising effect and image quality. In addition, the Gumbel sampling in the MIP loss can also be modified to other sampling methods that are sensitive to the maximum value distribution.

[0087] A 3D-TOF-MRA image generation model training method provided in this embodiment uses three easily available MRI images, T1-Weig hted, T2-weighted and Flair, to train a model capable of generating high-quality 3D-TOF-MRA images. It uses the adversarial characteristics of the conditional generative adversarial network to enhance the generalization characteristics of the generative model, so that the model has good performance on a larger data range; the background network architecture of the generator is preferably Unet++, which introduces multi-level path connections and dense jump connections. Through these connection mechanisms, the efficiency of information propagation and path selection in the network are improved, which helps to better integrate and utilize multi-scale features. In addition, considering that MIP images are often used to observe blood vessels in diagnosis, this embodiment introduces MIP loss based on Gumbel random sampling to improve the quality of the generated 3D-TOF-MRA images. In 3D-TOF-MRA images, blood vessels present high signals. Gumbel sampling can approximately simulate the distribution of high signals in the image, thereby strengthening the constraints on the high signal areas, which are clinically important parts. Therefore, it can increase the model's attention to vascular details in 3D-TOF-MRA images while generating basic structures, thereby improving the quality of the generated images and the accuracy of the blood vessels.

[0088] Compared with the prior art that enhances low-dose 3D-TOF-MRA images or uses single-modality images to generate 3D-TOF-MRA images, this embodiment uses three modality images with different characteristics to generate 3D-TOF-MRA images with higher quality and more precise details, which is of great significance for accurate lesion positioning and diagnosis in clinical practice.

[0089] Embodiment 2

[0090] See also Figure 6 , this embodiment provides a 3D-TOF-MRA image generation model training device, including:

[0091] A data acquisition module is used to acquire T1-weighted images, T2-weighted images, Flair images and TOF-MRA standard images of the same part;

[0092] A data preprocessing module, used for preprocessing the T1-weighted image, the T2-weighted image, the Flair image and the TOF-MRA standard image;

[0093] A generator module, used for splicing the pre-processed T1-weighted image, the T2-weighted image and the Flair image, and inputting them into a generator of a pre-built conditional generative adversarial network to obtain a generator output image; wherein the generator is a Unet architecture network;

[0094] A discriminator module, used to splice the pre-processed T1-weighted image, the T2-weighted image and the Flair image with the TOF-MRA standard image, and input them into the discriminator of the pre-built conditional generative adversarial network to obtain a real discriminant image output by the discriminator;

[0095] and concatenating the preprocessed T1-weighted image, the T2-weighted image and the Flair image with the generator output image, and inputting the concatenated images into the discriminator to obtain a generated discriminant image output by the discriminator;

[0096] A model training module, used to calculate the loss function between the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and back-propagate to update the discriminator weights;

[0097] And calculate the loss function between the generator output image and the TOF-MRA standard image and the loss function between the generated discriminant image and the all-1 matrix, and back-propagate to update the generator weights;

[0098] And repeatedly updating the discriminator weight and the generator weight until a model capable of generating high-quality 3D-TOF-MRA images is trained.

[0099] For the specific implementation content of each module in a 3D-TOF-MRA image generation model training device, please refer to the above definition of a 3D-TOF-MRA image generation model training method, which will not be repeated here.

[0100] The technical features of the above embodiments may be arbitrarily combined (as long as there is no contradiction in the combination of these technical features). To make the description concise, not all possible combinations of the technical features in the above embodiments are described; these embodiments that are not explicitly written should also be considered to be within the scope of this specification.

Claims

1. A 3D-TOF-MRA image generation model training method, characterized in that: include: Step 1: Obtain T1-weighted images, T2-weighted images, Flair images, and TOF-MRA standard images of the same part; Step 2: preprocessing the T1-weighted image, the T2-weighted image, the Flair image and the TOF-MRA standard image; Step 3: splicing the preprocessed T1-weighted image, the T2-weighted image and the Flair image with the TOF-MRA standard image, and inputting them into the discriminator of the pre-built conditional generative adversarial network to obtain the real discriminant image output by the discriminator; Step 4: splicing the preprocessed T1-weighted image, the T2-weighted image and the Flair image, and inputting them into a generator of a pre-built conditional generative adversarial network to obtain a generator output image; wherein the generator is a Unet architecture network; Step 5: splicing the preprocessed T1-weighted image, the T2-weighted image and the Flair image with the generator output image, and inputting them into the discriminator to obtain a generated discriminant image output by the discriminator; Step 6: Calculate the loss function between the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and back-propagate to update the discriminator weights; Step 7: Calculate the loss function between the generator output image and the TOF-MRA standard image and the loss function between the generated discriminant image and the all-1 matrix, and back-propagate to update the generator weights; Step 8: Repeat steps 6 and 7 until a model capable of generating high-quality 3D-TOF-MRA images is trained.

2. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that: Step 2 specifically includes: registering the T1-weighted image, the T2-weighted image and the Flair image according to the TOF-MRA standard image, and normalizing the T1-weighted image, the T2-weighted image, the Flair image and the TOF-MRA standard image.

3. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that: In step 3, the discriminator adopts a multi-layer convolutional block stacking structure.

4. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that: In step 3, the discriminator adopts mean square error loss.

5. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that: In step 4, the Unet architecture network is Unet++, Attention U-Net or Mamba-UNet.

6. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that: In step 4, the basic module of the generator is VGG_Block, which includes instance normalization, convolutional spectral normalization and a dropout layer.

7. The 3D-TOF-MRA image generation model training method according to claim 6, characterized in that: The activation function of the VGG_Block adopts LeakReLU.

8. The 3D-TOF-MRA image generation model training method according to claim 1, characterized in that: In step 4, the generator adopts L1 loss, structural similarity loss and maximum projection mixed loss.

9. The 3D-TOF-MRA image generation model training method according to claim 8, characterized in that: The maximum projection mixed loss is a maximum projection mixed loss based on Gumbel random sampling.

10. A 3D-TOF-MRA image generation model training device, characterized in that: include: A data acquisition module is used to acquire T1-weighted images, T2-weighted images, Flair images and TOF-MRA standard images of the same part; A data preprocessing module, used for preprocessing the T1-weighted image, the T2-weighted image, the Flair image and the TOF-MRA standard image; A generator module, used for splicing the pre-processed T1-weighted image, the T2-weighted image and the Flair image, and inputting them into a generator of a pre-built conditional generative adversarial network to obtain a generator output image; wherein the generator is a Unet architecture network; A discriminator module, used to splice the pre-processed T1-weighted image, the T2-weighted image and the Flair image with the TOF-MRA standard image, and input them into the discriminator of the pre-built conditional generative adversarial network to obtain a real discriminant image output by the discriminator; and concatenating the preprocessed T1-weighted image, the T2-weighted image and the Flair image with the generator output image, and inputting the concatenated images into the discriminator to obtain a generated discriminant image output by the discriminator; A model training module, used to calculate the loss function between the real discriminant image and the all-1 matrix and the generated discriminant image and the all-0 matrix, and back-propagate to update the discriminator weights; And calculate the loss function between the generator output image and the TOF-MRA standard image and the loss function between the generated discriminant image and the all-1 matrix, and back-propagate to update the generator weights; And repeatedly updating the discriminator weight and the generator weight until a model capable of generating high-quality 3D-TOF-MRA images is trained.

Images