Deep learning method for magnetic resonance image optimization and generation based on multi-contrast complementation

Abstract

The present invention belongs to a medical image processing method and relates to the technical field of deep learning. The present invention provides a deep learning method for optimizing and generating magnetic resonance images based on multi-contrast complementation, and generates 7T-like or 3D-like contrast magnetic resonance images from source contrast 3T or 2D magnetic resonance images. The method comprises the following steps: data preprocessing, model construction, training a generative adversarial network, reasoning, and data postprocessing. The present invention fully utilizes complementary information between magnetic resonance images of different contrasts, and the deep learning method for optimizing and generating magnetic resonance images based on multi-contrast complementation successfully realizes the enhancement of accelerated 3T and 2D magnetic resonance images, thereby generating high-quality 7T-like and 3D-like image quality. The method not only improves the efficiency and quality of clinical scanning, but also provides the possibility for more convenient use of 7T-like and 3D magnetic resonance images in clinical practice.

Description

Technical Field

[0001] The present invention belongs to a medical image processing method and relates to the technical field of deep learning, and specifically to a deep learning method for magnetic resonance image optimization and generation based on multi-contrast complementarity. Background Art

[0002] High signal-to-noise ratio magnetic resonance (MR) imaging (3D MR and 7T MR) has higher imaging resolution and signal-to-noise ratio (SNR) than low signal-to-noise ratio MR scanning (2D MR and 3T MR), which can better describe the structure, function, metabolism and pathology of human tissues, thus helping doctors to make more accurate clinical diagnoses. However, the limited clinical accessibility of high signal-to-noise ratio scanners has limited their widespread use in clinical applications.

[0003] Deep learning-based synthesis methods use convolutional neural networks (CNNs) to learn the nonlinear mapping between source and target images end-to-end, showing higher accuracy than traditional methods, but still have many limitations.

[0004] Existing methods usually establish a direct mapping relationship from the source image to the target image, or learn the data distribution of the target image from random Gaussian noise. When the source image and the target image are very different, the synthesis is difficult, the generated image is blurred, and it is different from the real image; existing methods usually train models on natural images, lack of information capture between different slices of multi-contrast medical images, and the residual blocks in the network used usually do not fully utilize the information between multiple channels; existing methods mostly use L1 loss, SSIM loss and their combination as loss functions to train the network, which is lacking in TOF image reconstruction, especially MIP image reconstruction based on TOF images.

[0005] In summary, the shortcomings of the above methods will cause important features (for brain MRI, such as blood vessels in T1 images, arterioles in TOF images, etc.) to be blurred or too smooth in the synthesized images, and the similarity between the synthesized images and the real acquired images is low. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a deep learning method for magnetic resonance image optimization and generation based on multi-contrast complementarity, which enhances T1-weighted (T1w) magnetization-prepared rapid gradient echo acquisition and time-of-flight (TOF) scanning through a novel multi-contrast deep learning framework, aiming to bridge the resolution and contrast gap between low signal-to-noise ratio and high signal-to-noise ratio magnetic resonance images, and simultaneously achieve 7T-level or 3D-level quality synthesis of T1w and TOF images to solve the problems mentioned in the background technology.

[0007] A deep learning method for optimizing and generating magnetic resonance images based on multi-contrast complementarity is used to generate a high signal-to-noise ratio magnetic resonance image from a source contrast low signal-to-noise ratio magnetic resonance image, including the following steps:

[0008] S1. Data preprocessing: S101. Obtain two contrast MRI images of the same part with low signal-to-noise ratio and high signal-to-noise ratio, convert the dicom format image into nifti format, and use the low signal-to-noise ratio image as the source image and the high signal-to-noise ratio image as the target contrast image;

[0009] S102, registering the remaining contrast images with the high signal-to-noise ratio TOF image respectively so that they are spatially aligned;

[0010] S103 , normalizing the above images, adjusting the data range to between 0 and 1, and obtaining a pre-processed source contrast image and a target contrast image.

[0011] S2. Model construction: construct a generative adversarial network to learn the mapping between source contrast image and target contrast image. The basic components of the generative adversarial network are the generator and the discriminator.

[0012] S3. Training the Generative Adversarial Network: A joint loss function consisting of structural similarity loss, L1 loss, discriminator loss, and maximum intensity projection loss is used. During training, the loss is calculated for the model output and target contrast image, and back-propagated to update the model parameters. The joint loss function is expressed as:

[0013] l total =λ1·l1+λ2·l SSIM +λ3·l MIP +λ4·l D

[0014] Among them, λ1, λ2, λ3, and λ4 represent the balance coefficients of L1 loss, structural similarity loss, maximum intensity projector, and discriminator loss, and their values ​​are 100, 100, 10, and 1, respectively;

[0015] S4, reasoning: the source contrast image is input into the model, the model synthesizes the target contrast image, no loss is calculated, and the model outputs a high signal-to-noise ratio image;

[0016] S5. Data post-processing: The value range of the model output is between 0 and 1. After restoring it to the original value range, use the save_as function of pydicom to save it in dicom format to obtain the final synthesis result.

[0017] Preferably, the generator uses a deep residual U-type network with multiple distillation information blocks, and the basic building blocks of the network are multiple distillation information blocks.

[0018] Preferably, the multiple distillation information block first performs feature extraction through four 3×3 convolution blocks, splits the channel after each convolution operation, uses one part for the next set of convolution operations, and concatenates the other part with the progressively refined features for channel splicing, extracts multi-contrast and progressively refined feature groups through the contrast-aware channel attention module, and then obtains a residual through a 1×1 convolution and adds it to the original feature.

[0019] Preferably, the contrast-aware channel attention module consists of a contrast information extraction layer, two convolutional layers, and a Sigmod activation function, and the contrast information extraction layer performs a sum operation on the mean and standard deviation of the input feature map.

[0020] Preferably, the discriminator determines whether each generated image belongs to a real high signal-to-noise ratio image category or an input low signal-to-noise ratio image category by increasing the loss.

[0021] Preferably, when training the generative adversarial network using a joint loss function consisting of structural similarity loss, L1 loss, discriminator loss and maximum intensity projection loss, Gumbel-softmax is used to approximate the maximum value of each pixel, where the MIP image calculation expression of the generated image is:

[0022]

[0023] Among them, temp is the temperature coefficient, which controls the hardness of the sampling. A larger temperature value will make the sampling softer, that is, closer to a uniform distribution, while a smaller temperature value will make the sampling harder and more inclined to a one-hot distribution. It is set to 5; g is the noise sampled from the Gambel distribution.

[0024] Preferably, when training the generative adversarial network, the input images are n consecutive TOF image slices and n corresponding T1 image slices, and the training is stopped when the number of training rounds reaches 200.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention makes full use of the complementary information between multi-contrast magnetic resonance images and proposes a deep learning method based on multi-contrast complementary magnetic resonance image optimization and generation, which successfully realizes the enhancement of low signal-to-noise ratio magnetic resonance images to generate high-quality high-signal-to-noise ratio image quality. It not only improves the efficiency and quality of clinical scanning of low signal-to-noise ratio images, but also provides the possibility for more convenient use of high-signal-to-noise ratio magnetic resonance in clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A generative adversarial network for the present invention;

[0028] Figure 2 The multiple distillation information block of the present invention;

[0029] Figure 3 The contrast perception channel attention module of the present invention;

[0030] Figure 4 It is a partial case display diagram of TOF and its corresponding MIP image of the present invention;

[0031] Figure 5 This is a partial case display diagram of the T1 image of the present invention;

[0032] Figure 6 It is a display diagram of the image quality evaluation index of the present invention;

[0033] Figure 7 This is a partial case display diagram of the enhanced 2D T1 image of the present invention;

[0034] Figure 8 This is a partial case display diagram of the enhanced 2D T2 image of the present invention. DETAILED DESCRIPTION

[0035] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0036] As attached Figure 1 To Attachment Figure 8 As shown:

[0037] Embodiment 1: The present invention provides a deep learning method for optimizing and generating magnetic resonance images based on multi-contrast complementarity, generating a quasi-7T or quasi-3D contrast magnetic resonance image from a source contrast 3T or 2D magnetic resonance image, comprising the following steps:

[0038] S1. Data preprocessing: S101. Obtain two contrast MRI images of the same part with low signal-to-noise ratio and high signal-to-noise ratio, convert the dicom format image into nifti format, and use the low signal-to-noise ratio image as the source image and the high signal-to-noise ratio image as the target contrast image;

[0039] S102, registering the remaining contrast images with the high signal-to-noise ratio TOF image respectively so that they are spatially aligned;

[0040] S103 , normalizing the above images, adjusting the data range to between 0 and 1, and obtaining a pre-processed source contrast image and a target contrast image.

[0041] S2. Model construction: construct Figure 1 The generated adversarial network shown in FIG. 1 is used to learn the mapping between the source contrast image and the target contrast image. The basic components of the generated adversarial network are the generator and the discriminator.

[0042] S3. Training the Generative Adversarial Network: A joint loss function consisting of structural similarity loss, L1 loss, discriminator loss, and maximum intensity projection loss is used. During training, the loss is calculated for the model output and target contrast image, and back-propagated to update the model parameters. The joint loss function is expressed as:

[0043] l total =λ1·l1+λ2·l SSIM +λ3·l MIP +λ4·l D

[0044] Among them, λ1, λ2, λ3, and λ4 represent the balance coefficients of L1 loss, structural similarity loss, maximum intensity projector, and discriminator loss, and their values ​​are 100, 100, 10, and 1, respectively;

[0045] S4, reasoning: the source contrast image is input into the model, the model synthesizes the target contrast image without calculating the loss, and the model outputs a high signal-to-noise ratio image;

[0046] S5. Data post-processing: The value range of the model output is between 0 and 1. After restoring it to the original value range, use the save_as function of pydicom to save it in dicom format to obtain the final synthesis result.

[0047] The generator uses a deep residual U-network with multiple distilled information blocks, and the basic building block of the network is the multiple distilled information blocks.

[0048] like Figure 2As shown in the figure, the multi-distillation information block first extracts features through four 3x3 convolution blocks. After each convolution operation, the channel is split, one part is used for the next set of convolution operations, and the other part is spliced ​​with the progressively refined features. The multi-contrast and progressively refined feature groups are extracted through the contrast-aware channel attention module, and then a residual is obtained through a 1x1 convolution and added to the original feature. The multi-distillation information block used effectively utilizes the information of different channels, so that the details of the generated image are more complete.

[0049] like Figure 3 As shown in the figure, the contrast-aware channel attention module consists of a contrast information extraction layer, two convolutional layers, and a Sigmod activation function. The contrast information extraction layer sums the mean and standard deviation of the input feature map to generate information about the structure, texture, and edge, thereby enhancing image details.

[0050] Existing methods usually establish a direct mapping relationship from a source image to a target image, or learn the data distribution of a target image from random Gaussian noise. When the source image and the target image are very different, the synthesis difficulty is high, and the generated image is blurred and different from the real image. The network proposed in the present invention uses a discriminator to determine whether each generated image belongs to a high signal-to-noise ratio image category or an input low signal-to-noise ratio image category by increasing the loss, so that the generated image is closer to the contrast and detail structure of the real image.

[0051] Since the maximum intensity projection in the channel dimension involves the maximum value operation, which is not differentiable in the back propagation of deep learning, the joint loss function consisting of structural similarity loss, L1 loss, discriminator loss and maximum intensity projection loss is adopted to train the generative adversarial network, and Gumbel-softmax is used to approximate the maximum value of each pixel. The MIP image calculation expression of the generated image is:

[0052]

[0053] Among them, temp is the temperature coefficient, which controls the hardness of the sampling. A larger temperature value will make the sampling softer, that is, closer to a uniform distribution, while a smaller temperature value will make the sampling harder and more inclined to a one-hot distribution. It is set to 5; g is the noise sampled from the Gambel distribution.

[0054] When training the generative adversarial network, the input images are n consecutive TOF image slices and n corresponding T1 image slices. The training is stopped when the number of training rounds reaches 200.

[0055] Experimental example: Compared with single contrast composite images, multi-contrast composite images qualitatively provide more accurate vascular ( Figure 4 ) and organizations ( Figure 5 ) structure information.

[0056] exist Figure 4 In the figure, the first row shows a TOF image slice, while the other rows show the maximum intensity projection (MIP) of the TOF image volume. The box highlights the area where the advantages of the present invention can be demonstrated. The multi-contrast composite image can restore more detailed and accurate structural information such as blood vessels and arterioles. When T1w structural information is integrated in the method proposed in the present invention, more arterioles can be accurately restored.

[0057] Figure 5 The quality of T1 images was compared. When combined with TOF image information in the present invention, better 7T-like T1 contrast and tissue structure information can be restored. The multi-contrast composite image is very similar to the real 7T image in terms of T1 contrast and structural details.

[0058] like Figure 6 As shown in the figure, a series of comparisons were made between the proposed technique and single-contrast synthetic images, using peak signal-to-noise ratio (PSNR), structural similarity (SSIM), mean absolute error (MAE), and root mean square error (RMSE) as evaluation indicators. Quantitative comparison of image indicators shows that multi-contrast synthetic images significantly improve the quality of T1w and TOF sequences compared to single-contrast images. Figure 6 The quantitative comparison of image quality metrics is summarized, and the median and interquartile range of the scores are also provided. The proposed method performs better on all metrics, which proves the effectiveness of incorporating complementary image contrast information. In addition, according to paired t-test and Wilcoxon rank sum test, all p values ​​are << 0.01, indicating statistical significance.

[0059] Figure 7 , Figure 8 The results of high signal-to-noise ratio TOF assisting low signal-to-noise ratio T1 and T2 to generate high-resolution T1 and T2 are compared. In every two rows, the first row shows the high-resolution T1 and T2 image slices, low-resolution T1 and T2 image slices, and the results of the images generated by each model. The box highlights the area that can demonstrate the advantages of the present invention, and the lower right corner is the enlarged result. The first column of the second row shows the corresponding TOF image slices and the error rate graph. Figure 7 In the present invention, the method not only enhances the visual quality, but also restores the details of the tissue structure more accurately. Figure 8In the MRI, only the method of the present invention can restore the small high signal points (suspected multiple sclerosis lesions) in the high signal-to-noise ratio T2 image. In the sagittal plane, the clarity and accuracy of the restoration by the method of the present invention are significantly better than those of other methods.

[0060] Embodiment 2: This embodiment is basically the same as the previous embodiment, except that: in the data preprocessing step, other methods can be used for image registration, such as image registration methods based on mutual information, Fourier, wavelet transform and deep learning; other brain extraction methods can be used; the image can be normalized to between -1 and 1;

[0061] The feature extraction method of multi-contrast input images can be modified, and a feature extractor can be constructed for T1w and TOF images respectively and then feature fusion can be performed; other discriminator structures can be used;

[0062] During the model training step, different balancing coefficients can be used for the loss function.

[0063] Additionally, in order to provide a concise description of exemplary embodiments, all features of an actual embodiment (ie, those features that are not relevant to the best mode presently contemplated for carrying out the invention or those that are not relevant to implementing the invention) may not be described.

[0064] It will be appreciated that in the development of any actual implementation, as in any engineering or design project, numerous implementation-specific decisions may be made. Such a development effort may be complex and time-consuming, but will be a routine task of design, fabrication, and production for those of ordinary skill having the benefit of this disclosure without undue experimentation.

[0065] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A deep learning method for magnetic resonance image optimization and generation based on multi-contrast complementation, characterized in that: Generating a high signal-to-noise ratio magnetic resonance image from a low signal-to-noise ratio magnetic resonance image comprises the following steps: S1. Data preprocessing: S101. Obtain two contrast MRI images of the same part with low signal-to-noise ratio and high signal-to-noise ratio, convert the dicom format image into nifti format, and use the low signal-to-noise ratio image as the source image and the high signal-to-noise ratio image as the target contrast image; S102, registering the remaining contrast images with the high signal-to-noise ratio TOF image respectively so that they are spatially aligned; S103, normalizing the above images, adjusting the data range to between 0 and 1, and obtaining a pre-processed source contrast image and a target contrast image; S2. Model construction: A generative adversarial network is constructed to learn the mapping between source contrast images and target contrast images. The basic components of the generative adversarial network are the generator and the discriminator. The generator uses a deep residual U-type network with multiple distilled information blocks. The basic components of the network are multiple distilled information blocks. The multiple distilled information blocks first extract features through four 3×3 convolution blocks. After each convolution operation, the channel is split, one part is used for the next set of convolution operations, and the other part is channel-joined with the progressively refined features. The contrast-aware channel attention module extracts multi-contrast and progressively refined feature groups, and then a residual is obtained through a 1×1 convolution and added to the original feature. The contrast-aware channel attention module consists of a contrast information extraction layer, two convolution layers, and a Sigmod activation function. The contrast information extraction layer sums the mean and standard deviation of the input feature map. S3. Training the Generative Adversarial Network: A joint loss function consisting of structural similarity loss, L1 loss, discriminator loss, and maximum intensity projection loss is used. During training, the loss is calculated for the model output and target contrast image, and back-propagated to update the model parameters. The joint loss function is expressed as: l total =λ1·l1+λ2·l SSIM +λ3·l MIP +λ4·l D Among them, λ1, λ2, λ3, and λ4 represent the balance coefficients of L1 loss, structural similarity loss, maximum intensity projector, and discriminator loss, and their values ​​are 100, 100, 10, and 1, respectively; S4, reasoning: the source contrast image is input into the model, the model synthesizes the target contrast image, no loss is calculated, and the model outputs a high signal-to-noise ratio image; S5. Data post-processing: The value range of the model output is between 0 and 1. After restoring it to the original value range, use the save_as function of pydicom to save it in dicom format to obtain the final synthesis result.

2. The deep learning method for optimizing and generating magnetic resonance images based on multi-contrast complementarity as claimed in claim 1 is characterized by: The discriminator determines whether each generated image belongs to the category of real high signal-to-noise ratio image or the category of input low signal-to-noise ratio image by increasing the loss.

3. The deep learning method for optimizing and generating magnetic resonance images based on multi-contrast complementarity as claimed in claim 1, characterized in that: When training the generative adversarial network using a joint loss function consisting of structural similarity loss, L1 loss, discriminator loss, and maximum intensity projection loss, for the task of generating 7T images from 3T images, Gumbel-softmax is used to approximate the maximum value of each pixel, where the MIP image calculation expression of the generated image is: Among them, temp is the temperature coefficient, which controls the hardness of the sampling. A larger temperature value will make the sampling softer, that is, closer to a uniform distribution, while a smaller temperature value will make the sampling harder and more inclined to a one-hot distribution. It is set to 5; g is the noise sampled from the Gambel distribution.

4. The deep learning method for optimizing and generating magnetic resonance images based on multi-contrast complementarity as claimed in claim 1, characterized in that: When training the generative adversarial network, the input images are n consecutive TOF image slices and n corresponding T1 image slices. The training is stopped when the number of training rounds reaches 200.

Images