DCE-MRI image generation method based on multi-constrained GAN

Abstract

The present invention discloses a DCE-MRI image generation method based on multi-constrained GAN, in which the first phase enhanced image I1 after contrast agent injection is generated. ’ Since there are no other enhanced images of adjacent phases, there is no constrained image at this time. The generator in the multi-constrained GAN only uses the image I0 without contrast agent injection to generate the enhanced image I1 ’ . After generating the enhanced image I at the nth time phase after contrast agent injection n ’ (n>1), the generated enhanced image I1 ’ Together with the image I0 without contrast agent injection, it forms a two-channel multi-constrained image. At this time, the generator in the multi-constrained GAN is a multi-constrained neural network, which will make full use of the attenuation information of the contrast agent at different time phases to generate the enhanced image I1 ’ .

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a DCE-MRI image generation method based on multi-constrained GAN. Background Art

[0002] In recent years, generative models that generate new samples by learning the probability density distribution of known samples have attracted widespread attention. The objective function of the generative model is the distance between the data distribution and the model distribution, which can be solved by the maximum likelihood method. However, it is usually difficult to solve it directly using the maximum likelihood method. However, Generative Adversarial Networks (GAN) uses the learning ability of neural networks to fit the distance between two distributions, which can cleverly avoid the difficulty of solving the likelihood function. It is currently the most successful and widely used generative model.

[0003] Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) images can obtain continuous dynamic enhanced images of tissues before, during, and after the injection of contrast agents, thereby reflecting the physiological metabolic changes of lesion tissues. However, patients who are allergic to injected contrast agents or have poor heart, liver, lung, and kidney function cannot undergo DCE-MRI examinations, and the lesions cannot be displayed more effectively. If DCE-MRI images can be generated based on plain MRI images using GAN instead of multiple MRI scans after contrast agent injection, it can not only provide the physiological metabolic changes of the patient's lesion tissues, but also do not need to inject contrast agents for secondary MRI dynamic scans.

[0004] Although no researchers have proposed a GAN structure for DCE-MRI image generation, GAN or convolutional neural network (CNN) has been widely used to generate cross-modal MRI images. Chartsias et al. used U-Net to extract effective features of MRI images of different modalities, and then fused these features to generate images of new modalities. This method was used to generate T2 and FLAIR images based on MRI T1 images in the BRATS 2015 database, and to generate FLAIR images based on ISLES MRI T1, T2 and DWI images. Yu et al. proposed Edge-Aware GANs, which consists of a generator, a discriminator and an edge detection module. The generator and the discriminator are used to generate images of new modalities and judge the authenticity of the generated new images, respectively. The edge detection module adds the edge information of the image to the GAN in the form of a loss function, so that the three modules can interact with each other. The network generates corresponding T2 and FLAIR images based on MRI T1 images in the BRATS 2015 database, and generates T2 images based on MRI PD images in IXI images. The generation results are satisfactory. Dar et al. [3] proposed a conditional GAN ​​to achieve the conversion between MRI T1 and T2 images, and used a perceptual loss function to focus on the details of the generated image. Yang et al. proposed a semi-supervised GAN to achieve the conversion of MRI images between two modalities. The supervised network ensures the accuracy of the space between the generated image and the original image, while the unsupervised network provides the generated image with a realistic visual effect that is significantly different from the original image.

[0005] At present, all existing methods process conventional contrast-enhanced magnetic resonance imaging (CE-MRI) rather than DCE-MRI. CE-MRI is a single imaging after the injection of contrast agent, generating a three-dimensional image. DCE-MRI is multiple imaging after the injection of contrast agent, which is a four-dimensional time series image. Therefore, compared with CE-MRI, there is more useful correlation information between DCE-MRI at different phases. Therefore, multi-constrained GAN makes full use of the correlation information between enhanced images at different phases to generate DCE-MRI images. Summary of the invention

[0006] In view of this, the present invention discloses a DCE-MRI image generation method based on multi-constraint GAN, so as to fully utilize the constraint relationship between images of different phases to generate four-dimensional DCE-MRI images.

[0007] The technical solution provided by the present invention is specifically a DCE-MRI image generation method based on multi-constrained GAN, comprising the following steps:

[0008] S1: Acquire data to obtain image data of different phases, including image I0 obtained by scanning once before contrast agent injection, and images I1-I obtained by scanning multiple times at equal intervals after contrast agent injection. n ;

[0009] S2: Train a multi-constrained generative adversarial network. When generating an enhanced image I1', no enhanced image is generated, and only image I0 is used to predict the generation. n When (n>2), under the constraints of the scan image I0 before contrast agent injection and the predicted enhanced image I1', the conventional magnetic resonance image and the effect of contrast agent are learned to generate enhanced images I2'-I n ';

[0010] S3: Image I1'-I n 'Respectively with the real image I1-I n The input is sent to the discriminator of the generative adversarial network, which determines the authenticity of the input image.

[0011] S4: The training of the multi-constrained generative adversarial network is completed until the discriminator cannot distinguish whether its input image is a generated image or a real image.

[0012] The generator in the multi-constrained generative adversarial network is a U-Net structure. n ', the generator in the multi-constrained generative adversarial network can simultaneously learn the features of conventional magnetic resonance images and the effects of contrast agents under the constraints of image I0 and the generated enhanced image I1', and generate an enhanced image I n '.

[0013] When generating the first phase image after contrast agent injection, the input of the generator is the real image I0; when generating the nth phase image I n When '(n>1), the input of the generator is a two-channel image consisting of image I0 and enhanced image I1'. The generator in the multi-constrained generative adversarial network includes a contraction descent path and a symmetric expansion path. The contraction descent path extracts low-dimensional features of the image, and the expansion path extracts high-dimensional features of the image. The output of the convolutional layer on the contraction descent path is connected to the input of the convolutional layer on the expansion path through a jump connection.

[0014] According to the different resolutions of feature maps, the paths in the generator network in the multi-constrained GAN are divided into different states, where each state consists of 2 blocks, each block contains a 3×3 convolutional layer followed by a BN layer and a ReLu layer. On the contraction or expansion path, the second block is connected to a maximum downsampling or upconvolution layer with a kernel size of 2×2, so that the size of the feature map is halved or doubled in both directions.

[0015] According to the different resolutions of feature maps, the paths in the discriminator network in the multi-constrained generative adversarial network are divided into different states, including 4 states, each state consists of 2 blocks, each block contains a 3×3 convolution layer followed by a BN layer and a ReLu layer, a downsampling layer with a kernel size of 2×2 is connected after the second block, and the last layer is a 1×1 convolution layer, and the softmax activation function is used to classify the input.

[0016] The DCE-MRI image generation method based on multi-constraint GAN provided by the present invention generates DCE-MRI images by using constraint information between different time phases. The method can realize the generation of DCE-MRI images based on plain scan MRI images by using the proposed multi-constraint GAN to replace multiple MRI scanning operations after injection of contrast agent. It can provide physiological metabolic changes of patient lesion tissues without the need to inject contrast agent for secondary MRI dynamic scanning.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A multi-constrained GAN structure diagram provided for an embodiment disclosed in the present invention;

[0021] Figure 2 A structural diagram of a generator provided for an embodiment disclosed in the present invention;

[0022] Figure 3 A structural diagram of a discriminator provided in an embodiment disclosed in the present invention;

[0023] Figure 4 A breast DCE-MRI image representation is generated based on a multi-constrained GAN provided by the disclosed embodiment of the present invention. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

[0025] GAN mainly consists of a generator and a discriminator. The goal of the generator is to generate realistic pseudo samples that the discriminator cannot distinguish between true and false. The goal of the discriminator is to correctly distinguish whether the data is a real sample or a pseudo sample from the generator. When the discriminator's recognition ability reaches a certain level but cannot correctly determine the source of the data, the training of the GAN is completed. At present, the GAN network has been widely used in the field of medical image processing, but no researcher has proposed a GAN structure that simultaneously utilizes the mutual constraints between multi-phase images to generate DCE-MRI images.

[0026] To this end, this embodiment provides a DCE-MRI generation method based on multi-constrained GAN, including: S1: acquiring data to obtain image data of different phases, including scanning once before injection of contrast agent to obtain image I0, and scanning multiple times at equal intervals after injection of contrast agent to obtain images I1-I n ;

[0027] S2: Train a multi-constrained generative adversarial network. When generating an enhanced image I1', no enhanced image is generated, so only image I0 is used to predict the generation. n When (n>2), under the constraints of the scan image I0 before contrast agent injection and the predicted enhanced image I1', the conventional magnetic resonance image and the effect of contrast agent are learned to generate enhanced images I2'-I n ';

[0028] S3: Image I1'-I n 'Respectively with the real image I1-I n The input is sent to the discriminator of the generative adversarial network, which determines the authenticity of the input image.

[0029] S4: The training of the multi-constrained generative adversarial network is completed until the discriminator cannot distinguish whether its input image is a generated image or a real image.

[0030] This implementation scheme aims at the relationship between adjacent phase DCE-MRI images and proposes a GAN structure based on adjacent phase constraints.

[0031] The multi-constrained GAN structure is as follows Figure 1As shown in the figure, the weighted fat suppression image of breast DCE-MRI T1 phase is used as an example to illustrate the GAN structure. Image I0 is obtained by scanning once before contrast agent injection, and images I1-I5 are obtained by scanning five times at intervals of 90s after injection. The generated images are I2'-I n 'express.

[0032] When predicting and generating the first phase enhanced image I1', the input of the multi-constrained GAN is the two-dimensional MRI image I0 of the T0 phase before the injection of contrast agent; when predicting and generating the nth phase enhanced image I n '(n>2), the input of the multi-constrained GAN is a two-channel image consisting of I0 and the predicted T1 phase enhanced image I1'. The generator generates the T n MRI images of phase I n ', and then GAN I n 'With the real image I n The input is sent to the discriminator, which judges the authenticity of the input image. When the discriminator cannot distinguish whether the input image is a generated image or a real image, the training of the multi-constrained GAN is completed.

[0033] The generator in the multi-constrained generative adversarial network is a U-Net structure. n ', the generator in the multi-constrained GAN can simultaneously learn the features of conventional magnetic resonance images and the effects of contrast agents under the constraints of image I0 and the generated enhanced image I1', and generate an enhanced image I n '.

[0034] Specifically, the generator structure in the multi-constrained GAN is as follows: Figure 2 As shown. The generator in the multi-constrained generative adversarial network is a U-Net structure. n ', the generator in the multi-constrained generative adversarial network can simultaneously learn the features of conventional magnetic resonance images and the effects of contrast agents under the constraints of image I0 and the generated enhanced image I1', and generate an enhanced image I n ';

[0035] The multi-constrained generative adversarial network includes a shrinking descent path and a symmetric dilation path. The shrinking descent path extracts low-dimensional features of the image, and the dilation path extracts high-dimensional features of the image. The output of the convolutional layer on the shrinking descent path is connected to the input of the convolutional layer on the dilation path through a jump connection.

[0036] The generative network consists of a contraction descent path on the left and a symmetrical expansion path on the right, which extract low-dimensional and high-dimensional features of the image respectively. The output of the convolutional layer on the contraction path is connected to the input of the convolutional layer on the expansion path through a jump connection, so that the network can simultaneously use low-dimensional and high-dimensional features to generate images. Depending on the resolution of the feature map, the paths in the network can be divided into different states. Each state consists of 2 blocks, each of which contains a 3×3 convolutional layer followed by a BN layer and a ReLu layer (the BN layer and the ReLu layer are simplified and not shown in the figure). On the contraction (expansion) path, the second block is connected to a maximum downsampling (up-convolution) layer with a kernel size of 2×2, so that the size of the feature map is halved (doubled) in both directions.

[0037] The discriminator structure is as follows Figure 3 As shown in the figure, it adopts a typical CNN structure. According to the different feature map resolutions, the network structure can be divided into 4 states. Each state consists of 2 blocks, each of which contains a 3×3 convolutional layer followed by a BN layer and a ReLu layer ( Figure 3 (not shown in Figure 3). A downsampling layer with a kernel size of 2×2 is connected after the second block to halve the size of the feature map and extract features of different scales. The last layer is a 1×1 convolutional layer, which reduces the number of feature maps to 1 and uses a softmax activation function to classify the input.

[0038] When training the generator, the mean absolute error (MAE) is used as the loss function of the generator. MAE is used to measure the average error between the image generated by the sub-network and the gold standard image. When training the discriminator, the cross entropy function is used as the loss function. The cross entropy describes the distance between the actual output and the expected output. The smaller the cross entropy value, the closer the two probability distributions are, and the more accurate the classification result is.

[0039] The above method was applied to 60 patients' breast DCE-MRI T1-weighted fat-suppressed images (patient privacy information will be removed) as experimental data. Each patient was scanned once before injection of contrast agent (gadopentic acid dimeglumine) to generate image I0, and scanned five times at 90s intervals after injection to generate images I1-I5. The size of each set of images was 784×784×180 pixels, and its resolution in the X, Y and Z directions was 0.45mm, 0.45mm and 1.00mm respectively. The images were trilinearly interpolated to make their resolution in all three directions 1.00mm.

[0040] Figure 4 The figure shows a set of original images and prediction results of breast DCE-MRI images at different phases. The generated images at different phases can reflect the real breast tissue structure, and the generated results are satisfactory.

[0041] Experiments were conducted on 30 groups of images in the database, and the average peak signal-to-noise ratio between the generated images and the real images was used to evaluate the image generation quality. The peak signal ratios of the generated images in the five phases can reach 12.34, 13.16, 11.96, 14.54 and 13.21 respectively.

[0042] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the claims.

Claims

1. A DCE-MRI image generation method based on multi-constrained GAN, characterized in that: The steps include: S1: Acquire data to obtain image data of different phases, including image I0 obtained by scanning once before contrast agent injection, and images I1-I obtained by scanning multiple times at equal intervals after contrast agent injection. n ; S2: Training multi-constrained GAN; When generating the enhanced image I1', use image I0 for prediction; When generating the enhanced image I n ', where n>2, under the constraints of the scan image I0 before contrast agent injection and the predicted enhanced image I1', the conventional magnetic resonance image and the effect of contrast agent are learned to generate enhanced images I2'-I n '; S3: Image I1'-I n 'Respectively with the real image I1-I n The input is sent to the discriminator of the generative adversarial network, which determines the authenticity of the input image. S4: The training of the multi-constrained GAN is completed until the discriminator cannot distinguish whether its input image is a generated image or a real image; The generator in the multi-constrained GAN is a U-Net structure; the enhanced image I2'-I is generated n ', the generator in the multi-constrained GAN can simultaneously learn the features of conventional magnetic resonance images and the effects of contrast agents under the constraints of image I0 and the generated enhanced image I1', and generate an enhanced image I n '; When generating the first phase image after contrast agent injection, the input of the generator is the real image I0; when generating the nth phase enhanced image I n ’ When n>2, the input of the generator is a two-channel image consisting of image I0 and enhanced image I1'.

2. The DCE-MRI image generation method based on multi-constrained GAN according to claim 1 is characterized in that: The generator in the multi-constrained GAN includes a shrinking descent path and a symmetric expansion path, wherein the shrinking descent path extracts low-dimensional features of the image, and the expansion path extracts high-dimensional features of the image, and the output of the convolutional layer on the shrinking descent path is connected to the input of the convolutional layer on the expansion path through a jump connection.

3. The DCE-MRI image generation method based on multi-constrained GAN according to claim 2 is characterized in that: According to the different resolutions of feature maps, the paths in the generator network in the multi-constrained GAN are divided into different states, where each state consists of 2 blocks, each block contains a 3×3 convolutional layer followed by a BN layer and a ReLu layer. On the contraction or expansion path, the second block is connected to a maximum downsampling or upconvolution layer with a kernel size of 2×2, so that the size of the feature map is halved or doubled in both directions.

4. The DCE-MRI image generation method based on multi-constrained GAN according to claim 2 is characterized in that: According to the different resolutions of the feature maps, the paths in the discriminator network in the multi-constrained GAN are divided into different states, including 4 states, each state consists of 2 blocks, each block contains a 3×3 convolution layer followed by a BN layer and a ReLu layer, a downsampling layer with a kernel size of 2×2 is connected after the second block, and the last layer is a 1×1 convolution layer, and the softmax activation function is used to classify the input.

Images