Generation method of magnetic resonance gadolinium contrast enhanced image based on contrast learning

Abstract

The invention discloses a method for generating a magnetic resonance gadolinium contrast enhancement image based on contrast learning and an SDACL model, and further comprises the step of carrying out patch-based multi-layer contrast learning on the SDACL model and adjusting a designed loss function, the SDACL model comprises a domain X representing a multi-mode MRI channel superposition image, a domain Y representing a T1CE image, generators G and F and a discriminator network; according to the method, T1, T2 and Flair magnetic resonance plain scanning sequence images of a glioma patient are utilized to generate a T1 weighted enhanced sequence image, an attention dilation generator is adopted, a receptive field is expanded by utilizing an expansion residual block, and local features and global dependency of an input image are combined by means of a convolutional attention module (CBAM). According to the method, the feature representation and context learning of the multi-sequence MRI are enhanced, a comprehensive loss function is further designed, the comprehensive loss function comprises patch-level contrast loss and structural similarity loss, noise can be effectively restrained, and the structural consistency of a generated image and a real image is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of nuclear magnetic resonance image generation, and in particular, relates to a method for generating a magnetic resonance gadolinium contrast-enhanced image based on contrast learning. Background Art

[0002] Gadolinium-based contrast agent (GBCA) is an important agent in magnetic resonance imaging, used to enhance the contrast of MRI images. After the patient is injected with contrast agent, an imaging scan is performed. GBCA enters the normal and abnormal tissues of the human body through the blood circulation, resulting in differences in material distribution and metabolism between diseased tissues and normal tissues. This difference enables T1CE images to observe the distribution of contrast agents between tissues, thereby providing clear tumor boundaries and vascular supply. Although GBCA plays an important role in MRI imaging, its expensive use cost and potential safety hazards, such as nephrogenic systemic fibrosis, gadolinium deposition in the brain and bones, and allergic reactions, cannot be ignored. For patients with gliomas who require long-term MRI follow-up, reducing or avoiding the use of GBCA can help significantly reduce the associated risks. Studies have tried to reduce the injection dose of injected GBCA to reduce risks, but low-dose contrast-enhanced MRI images often cannot fully display the subtle features of lesions and cannot meet the clinical image quality requirements.

[0003] In recent years, deep learning technology has attracted much attention due to its significant potential in reducing or eliminating GBCA doses. Although deep learning models have shown great potential in MRI image generation, the information provided by relying solely on a single sequence of images is usually limited, which to some extent limits the performance of the model in accurately generating images. In contrast, multi-sequence images can provide more comprehensive and rich pathological information, which is crucial for doctors to accurately diagnose and precisely locate tumors. However, the information barriers between different sequences make it difficult for this information to be effectively integrated and utilized. In response to this challenge, the current research trend focuses on using contrastive learning strategies to assist the generator in learning contrastive representations between multimodal data, thereby promoting information fusion between different sequence images, and providing new possibilities for generating images with high quality and clinical diagnostic value.

[0004] In clinical practice, T1, T2 and Flair are three commonly used MRI plain scan sequences. With their unique imaging characteristics, they provide doctors with multi-angle pathological information, significantly improving the accuracy of clinical diagnosis. Inspired by contrastive learning, mapping existing multi-sequence images to the same representation space for learning can effectively fuse the features of multi-sequence MRI images to generate high-quality T1CE images. These generated images can be used to replace or assist real gadolinium contrast agent images, thereby significantly reducing or even avoiding the use of gadolinium contrast agents, effectively reducing the risk of patients receiving GBCA, and providing more reliable and valuable information for clinical diagnosis. Summary of the invention

[0005] In order to effectively reduce the dosage of GBCA while ensuring the imaging quality, the present invention provides a method for generating a magnetic resonance gadolinium contrast-enhanced image based on contrast learning, including an SDACL model, wherein the SDACL model includes a domain X representing a multimodal MRI channel superposition image and a domain Y representing a T1CE image;

[0006] Generators G and F are both expansion generators based on the attention mechanism module CBAM and dilated convolution introduced in the ResNet architecture. They are used to take T1CE images as input and generate enhanced images through the CBAM module;

[0007] The discriminator network uses the Markov discriminator PatchGAN as the discriminator network, which includes the first four convolutional layers for obtaining image features and the last convolutional layer for further processing the extracted features to provide suitable input for the fully connected layer;

[0008] It also includes patch-based multi-layer contrastive learning of the SDACL model and the adjustment of the designed loss function.

[0009] Furthermore, the dilation generator includes 2 downsampling blocks, 9 residual blocks and 2 upsampling blocks, each upsampling block and downsampling block follows a two-step convolution / deconvolution, normalization, ReLU structure, and each residual block includes dilated convolution, normalization, ReLU, dilated convolution, normalization, CBAM and residual connection.

[0010] It also includes 6 convolutional layers, and the convolution kernel sizes are 7×7, 3×3, 3×3, 3×3, 3×3 and 7×7, and the number of filters are 64, 128, 256, 128, 64 and 1 respectively.

[0011] Furthermore, the attention mechanism module CBAM includes a channel attention module CAM and a spatial attention module SAM, CAM is used to adaptively select important channel features and filter out unimportant channel information, and SAM is used to focus on the most useful spatial part of the image.

[0012] Furthermore, the Markov discriminator PatchGAN includes 5 convolutional layers, the first 4 convolutional layers are used to obtain image features, including four 4×4 convolutional layers, an instance normalization layer and a LeakyReLU activation function, the slope is set to 0.2, the slope of the activation function is set to 0.2, the step size of each convolutional layer is 2, the step size of the last layer is 1, and the output channels are 64, 128, 256 and 512 respectively. After the last convolutional layer, the discriminator uses a fully connected layer to output a one-dimensional predicted feature map to determine the degree of match between the input data and the given conditions.

[0013] Furthermore, the SDACL model uses PatchNCE loss to match the corresponding patches of the input image and the output image. Each feature represents a patch in the image. The corresponding features in the selected layer are used as positive samples, and other features are used as negative samples. The query patch, positive samples and negative samples are mapped to the K-dimensional vector space and expressed as:

[0014] v、v + ∈R K and v - ∈R N×K ;

[0015] Among them, v - ∈R N×K represents the nth negative sample patch;

[0016] Then, the vector is normalized using L2 normalization, and then an (N+1)-way classification problem is established to calculate the probability of selecting a positive sample instead of a negative sample, where the distance between the sample and other positive and negative samples is τ. The cross entropy loss function is used to indicate that the probability of a positive sample being selected is greater than the probability of a negative sample. The function expression is:

[0017]

[0018] Patch-based multi-layer contrastive learning maps the features of the input image and the generated image into the same embedding space, and the NCE loss forces the features of the generated T1CE to remain close to the features of patches in multiple plain scan sequences located at the same position, while staying away from other patches.

[0019] Furthermore, the loss function designed for the SDACL model includes an adversarial loss GANLoss for training the generator and the discriminator so that the images generated by the generator are closer and closer to the real images, a patch-level contrast loss PatchNCELoss for calculating the similarity between the generated images and the real images at the patch level, and a structural similarity loss SSIMLoss for measuring the structural similarity between the generated images and the real images, which helps to maintain the details and texture of the images.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] Using three MRI plain scan sequences of T1, T2 and Flair of patients with brain glioma, T1 weighted enhanced sequence images (T1CE) were generated. The attention expansion generator was used to expand the receptive field using the expansion residual block, and the local features of the input image were combined with its global dependency with the help of the convolutional attention module CBAM to strengthen the feature representation and contextual learning of multi-sequence MRI. In order to make the generated image closer to the target image in terms of structure and texture, a comprehensive loss function was designed, including patch-level contrast loss and structural similarity loss. This comprehensive loss function can effectively suppress noise and ensure the structural consistency of the generated image and the real image. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is an architectural diagram of the SDACL model in the present invention.

[0023] Figure 2 Schematic diagram of the structure of the attention mechanism module in the present invention.

[0024] Figure 3 Schematic diagram of the network structure of the generator in the present invention.

[0025] Figure 4 Schematic diagram of the structure of the discriminator network in the present invention.

[0026] Figure 5 It is a reference graph for multimodal patch-level contrastive learning in the present invention.

[0027] Figure 6 This is the visualization result of multiple sequence input in this embodiment.

[0028] Figure 7 This is a difference map between the image generated in this embodiment and the real image.

[0029] Figure 8 Graph showing pixel intensity in this embodiment.

[0030] Fig. 9 FIG. 4 is a schematic diagram of radiologists' scoring of images in this embodiment. DETAILED DESCRIPTION

[0031] In order to facilitate the understanding of the present invention, the device of the present invention will be described more fully below with reference to the relevant drawings. Embodiments of the device are given in the drawings. However, the device can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0032] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "set" should be understood in a broad sense, for example, they can be fixedly connected or set, or detachably connected or set, or integrally connected or set. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0033] Example

[0034] like Figure 1-2 As shown, this embodiment provides a method for generating a magnetic resonance gadolinium contrast-enhanced image based on contrast learning, including an SDACL model, the SDACL model includes a domain X representing a multimodal MRI channel superposition image and a domain Y representing a T1CE image, and includes two generators ( Figure 1 G and F are used in the representation), both of which are expansion generators based on the attention mechanism module CBAM and dilated convolution introduced in the ResNet architecture. Dilated convolution combined with residual learning is used to replace the residual block in the original generator, which can ensure that the generator obtains sufficient receptive field in the process of feature extraction and better captures the global context information of the input image. The enhanced image is generated after the CBAM module. Among them, the attention mechanism module CBAM can force the generator to pay attention to important features and ignore irrelevant details by learning the weight distribution based on the feature map. The attention mechanism module CBAM includes a channel attention module CAM and a spatial attention module SAM. CAM is used to adaptively select important channel features and filter out unimportant channel information. SAM is used to focus on the most useful spatial part of the image, such as enhancing the feature representation of the tumor area.

[0035] Specifically, generators G and F implement the mapping from domain X to domain Y: F implements the mapping from domain Y to domain X At the same time, generators G and F also include an encoder part G enc and F enc , and the decoder part G dec and F dec The structure of the encoder is used to convert the input image into a latent space representation, and the decoder is used to restore the latent space representation to the output image;

[0036] It also includes two discriminators ( Figure 1 D X , D Y denoted), and the discriminator network includes the first four convolutional layers for obtaining image features, and the last convolutional layer for further processing the extracted features to provide suitable input for the fully connected layer. A 70×70 Markov discriminator PatchGAN structure is used to judge overlapping image blocks of 70×70 size to determine whether they are real images or generated images. For each image block, the model runs a conventional discriminator to perform binary classification on the results, and finally feeds the discrimination results back to the generator to help the generator learn to generate realistic and detailed images;

[0037] It also includes patch-based multi-layer contrastive learning of the SDACL model and adjustment of the designed loss function;

[0038] and Figure 1 The embedding in the above code is an embedding representation that can map an image from a high-dimensional space to a vector representation in a low-dimensional space. For each mapping, image features are first extracted from the four-layer encoder and then sent to the two-layer MLP projection head H. X and H Y These projection heads can project the extracted features from the encoder into the feature stack and use them to measure similarity, calculate distance, or perform other post-processing tasks. Use EmbeddingX to get the feature vector of the real image X, and use EmbeddingY to get the synthetic image Then, the distance between the two feature vectors is measured by calculating NCELoss, thereby optimizing the generator network to generate more realistic images.

[0039] Detailed, such as Figure 3 As shown in the figure, the dilated generator includes 2 downsampling blocks, 9 residual blocks and 2 upsampling blocks. Each upsampling block and downsampling block follows the double-step convolution / deconvolution, normalization, and ReLU structure. Each residual block includes dilated convolution, normalization, ReLU, dilated convolution, normalization, CBAM and residual connection. It also includes 6 convolution layers, and the convolution kernel sizes are 7×7, 3×3, 3×3, 3×3, 3×3 and 7×7 respectively, and the number of filters is 64, 128, 256, 128, 64 and 1 respectively.

[0040] In detail, the discriminator network is a key component of GAN, which is used to evaluate the difference between the generated image and the real image. Markov discriminator PatchGAN is one of the commonly used discriminator networks. It is a convolutional network structure that can discriminate in the local area of ​​the image to capture more fine-grained image features. The Markov discriminator PatchGAN maps the input image to an n×n output matrix X, X ij represents the probability that each block is true, by ij The Markov discriminator PatchGAN divides the input image into n×n patches and makes independent judgments on each patch. Each True and False in the output matrix corresponds to an area of ​​the image. The final output is the average of all the regional results. This method allows the network to focus on each individual area instead of the entire image, thereby reducing the number of parameters and improving the accuracy of each area of ​​the generated image. Compared with ordinary discriminators, PatchGAN has the advantages of fewer parameters and faster operation speed.

[0041] like Figure 4 As shown in the figure, the Markov discriminator PatchGAN includes 5 convolutional layers. The first 4 convolutional layers are used to obtain image features, including four 4×4 convolutional layers, instance normalization layers and LeakyReLU activation functions. The slope is set to 0.2. The slope of the activation function is set to 0.2. The step size of each convolutional layer is 2, and the step size of the last layer is 1. The output channels are 64, 128, 256 and 512 respectively. After the last convolutional layer, the discriminator uses a fully connected layer to output a one-dimensional predicted feature map to determine the degree of match between the input data and the given conditions.

[0042] In detail, contrastive learning refers to learning data representation by maximizing the similarity between related samples and minimizing the similarity between unrelated samples. The SDACL model uses a noise contrast estimation framework to maximize the mutual information between the input image and the output image, and adopts PatchNCE loss to match the corresponding patches of the input image and the output image. Each feature represents a patch in the image. The corresponding features in the selected layer are used as positive samples, and other features are used as negative samples. The query patch, positive samples and negative samples are mapped to the K-dimensional vector space and expressed as:

[0043] v、v + ∈R K and v - ∈R N×K ;

[0044] Among them, v - ∈R N×K represents the nth negative sample patch;

[0045] Use L2 normalization to normalize the vector, and then establish an (N+1)-way classification problem, and calculate the probability of selecting a positive sample instead of a negative sample, where the distance between the sample and other positive and negative samples is τ, scaled by τ=0.07, and use the cross entropy loss function to indicate that the probability of a positive sample being selected is greater than the probability of a negative sample. The function expression is:

[0046]

[0047] The patch-based multi-layer contrastive learning maps the features of the input image and the generated image into the same embedding space, and the NCE loss forces the features of the generated T1CE to be close to the features of the patches in multiple plain scan sequences located at the same position, while being far away from other patches, such as Figure 5 As shown;

[0048] In detail, the loss function of the SDACL model consists of three terms: adversarial loss GANLoss, patch-level contrast loss PatchNCELoss, and structural similarity loss SSIMLoss, where X represents the input non-enhanced MRI sequence and Y represents the T1CE sequence;

[0049] Among them, the adversarial loss GANLoss uses adversarial loss to encourage the generator to generate images that are visually similar to images in the target domain. and the discriminator D Y , the GAN loss expression is as follows:

[0050] L GAN (G,D Y ,X,Y)=E y～Y [logD Y (y)]+E x～X [log(1-D Y (G(x)))];

[0051] In the above formula, G tries to generate images G(x) that look similar to images in domain Y, while D Y Aims to distinguish the generated G(x) from the real Y, mapping and the discriminator D x is similar to the GAN loss of mapping G;

[0052] Patch-level contrast loss PatchNCELoss uses G enc and H X Extract features from domain X and use F enc and H Y Extract features from domain Y and extract features from G enc (X) Select layer L and send it to H X , embedding the image into a bunch of features in represents the output of the lth selected layer, each feature represents a patch in the image, for the mapping The patch-based PatchNCE loss expression is as follows:

[0053]

[0054] In the above formula, represents the latent feature vector of T1CE generated by layer l in the n-th encoder at position s∈S, represents the characteristics of the corresponding positive sample, Represents the features of corresponding negative samples sampled from different locations in the same input image;

[0055] For the generated fake image G(x) belonging to domain Y, we take advantage of dual learning and use different embeddings of domain Y and invert the mapping Use similar losses;

[0056] Structural similarity loss SSIMLoss Using pixel-by-pixel loss alone may ignore the image structure. In order to better capture the structural information of the image, SSIM loss is added to generate more realistic images. SSIM is a model that perceives structural changes, which includes all information such as brightness, contrast, and structure. The SSIM loss expression is as follows:

[0057] L SSIM (X,Y)=1-SSIM(X,Y);

[0058] In the above formula, SSIM(X,Y) output is a scalar between 0 and 1, indicating the structural similarity between images X and Y; the overall loss function expression is as follows:

[0059]

[0060] In the above formula, λ GAN ,λ NCE and λ SSIM is a trade-off parameter used to balance the contribution of each loss term.

[0061] The steps of the experiment using the above-mentioned enhanced image generation method are as follows:

[0062] S1. Experimental preparation stage

[0063] S1-1. Experimental Dataset

[0064] A dataset of 402 brain glioma MRIs was used, each of which included axial T1, T2, Flair, and T1CE sequences (the dataset covered multiple devices of different models from different manufacturers, including 4 Siemens devices and 4 GE devices). The slice thickness of the T1 sequence ranged from 5.5 mm to 6.5 mm, the repetition time span ranged from 220 to 1970 milliseconds, and the echo time ranged from 2.46 to 28.60 milliseconds. The slice thickness of the T1CE sequence ranged from 5 mm to 6.5 mm, and the repetition time span ranged from 220 to 1970 milliseconds. The echo time ranged from 2.46 to 28.60 milliseconds. The repetition time span is 220 to 1970 milliseconds, the echo time is 2.46 to 28.60 milliseconds, the layer thickness of T2 and Flair sequences ranges from 4mm to 6.5mm, the repetition time span is 3000 to 5630 milliseconds, and the echo time is 90 to 118 milliseconds. At the same time, in the data preprocessing stage, those patient images that are obviously affected by edema or necrosis are screened out, and the images of T1, T2, and Flair sequences are used for multi-channel merging as input, and T1CE is used as output. To help train the model, all images are scaled to 256×256 and standardized to the interval [-1,1]. In addition, all data are randomly divided into training and test sets at a ratio of 4:1.

[0065] S1-2. Experimental setup

[0066] The SDACL model adopted follows the setting of DCLGNA (Dual Contrastive Learning GAN) for training, and the Adam optimizer is used to speed up the convergence speed, where the optimization parameters β1 are set to 0.5 and β2 are set to 0.999. In order to improve the stability and generation quality of the model, HingeGAN loss is used instead of LSGAN loss, and the 4th, 8th, 12th and 16th layers (i.e., the first and second downsampling convolutions of the encoder, and the first and fifth residual blocks) are selected to calculate the patch-level contrast loss. In the total loss function, the equalization parameters λGAN, λNCE and λSSIM are set to 1, 2 and 5 respectively. The specific value of λ is determined according to the experiment. The SDACL model has been trained for 200 cycles. The learning rate is kept at 0.0002 in the first 100 cycles, and the learning rate is gradually linearly decayed to 0.0001 in the next 100 cycles. The batch size is 4 and the instance normalization technique is used during training. All training images are loaded with a size of 286×286 and then scaled to 256×256.

[0067] The above model is implemented based on the Pytorch deep learning framework, and all experiments are performed on The results are performed on GoldCPUs 6330 CPU @ 2.00 GHz and 4 Tesla 4 GPUs (8 GB).

[0068] S1-3. Setting of evaluation method

[0069] In order to verify the effectiveness of the method proposed in this example, it was compared with four advanced cross-modal generation methods: CycleGAN, CUT, DCL, and Pix2pix. All methods followed the same experimental settings and performed qualitative and quantitative analysis. Finally, radiologists were invited to conduct subjective evaluation of the generated images.

[0070] (1) Qualitative evaluation In order to qualitatively evaluate the quality of the generated T1CE images, a visual analysis method was used to compare the differences between the input T1, T2, Flair and real T1CE images and the generated T1CE images. In order to more intuitively evaluate the differences between them, the difference map and pixel intensity curve between the real image and the generated image were also plotted. The value range of the difference map is [-1, 1], where -1 means completely similar, 1 means completely different, and 0 means irrelevant.

[0071] (2) Quantitative evaluation Three evaluation indicators widely used in the field of medical image generation are used to quantitatively evaluate the performance of the model, including the normalized root mean square error (NRMSE), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM).

[0072] (3) Subjective evaluation: Radiologists were invited to evaluate the similarity between the generated images and the real images.

[0073] A five-point scoring system was developed based on the similarities between the two, as shown in the following table:

[0074] Evaluation scale Fraction The generated images are almost indistinguishable from real images and can provide high-quality diagnostic information 5 The generated images are highly similar to the real images and can provide reliable diagnostic information. 4 There are some differences between the generated images and the real images, but they can still provide certain diagnostic information. 3 There is a significant difference between the generated images and the real images, making it difficult to provide accurate diagnostic information 2 The generated images are very different from the real images and cannot provide useful information 1

[0075] It should be noted that the scoring only considers the image and does not take into account other clinical data.

[0076] S1-4. Ablation experiment

[0077] In order to explore the importance of each key component in SDACL, this embodiment designs three degradation models to perform ablation experiments, as follows:

[0078] (1) In order to verify the key role of the dilated convolution in the model, it is replaced by the traditional convolution to obtain model A;

[0079] (2) In order to explore the impact of the attention mechanism block in the generator on the model performance, it is removed to obtain model B;

[0080] (3) In order to verify the impact of SSIM loss on model performance, it is removed to obtain model C.

[0081] The three downgraded models were compared with the full SDACL model in the experiment to evaluate the impact of key components on model performance. In addition, different sequence inputs were used to train the SDACL model to explore whether multi-sequence inputs help model performance.

[0082] S2. Experimental Results and Analysis

[0083] Qualitative analysis:

[0084] Figure 6 The visual comparison between the generated T1CE images and the real T1CE images obtained by different deep learning networks in the context of multi-sequence input is shown. The differences between the different generation methods can be clearly observed from the visualization results. Specifically, CUT's performance in generating images is not ideal. There are obvious perceptual differences between its generated images and real images, and the authenticity is obviously insufficient. The images generated by the CycleGAN method have certain visual deviations, and there are still large deviations in key details and features. The DCL and Pix2pix methods can better capture the contours and overall structure of the image, but their performance in details is still insufficient and they fail to accurately restore the tumor area. Compared with these four methods, the results generated by the SDACL model are closest to the real image. Not only is it similar to the real image in overall structure, but it also performs well in detailed textures and tumor areas. This is because the SDACL model combines SSIM loss with block-level contrast loss, which can better ensure the consistency of block content between the generated image and the real image, thereby improving the generator's ability to learn tumor features. In addition, in Figure 1 In the analysis, the tumor area of ​​the first case showed some subtle differences in the generated image, which may be related to the significant edema of the patient's lesion. Although the generated image is not completely consistent with the real image in details, by combining the features of T1, T2 and Flair sequences, it can still effectively capture and display the ring enhancement characteristics of the tumor. When combining multiple sequence images for comprehensive analysis, it helps doctors identify the exact location and morphology of the tumor during the diagnosis process and reduce the risk of misdiagnosis;

[0085] and Figure 7 The difference comparison between T1CE images generated by different networks and real T1CE images is shown. At the anatomical edge of the image, the limitations of networks such as CycleGAN, CUT, DCL and Pix2pix in the image generation process can be observed. These networks are insufficient in edge clarity and detail preservation in the tumor area, resulting in blurred edges and detail loss in the generated images. Compared with the above networks, the SDACL model shows significant advantages in image generation, outperforming other comparison networks in terms of accuracy and clarity of anatomical edges, and is closer to the essential characteristics of real images;

[0086] Figure 8 The curve comparison of pixel intensity between the generated image and the real image is depicted. In this image, the horizontal axis represents the pixel position and the vertical axis indicates the intensity value of the corresponding pixel point. Through comparative analysis, it can be observed that the SDACL model shows excellent performance in reconstructing tumor area blocks, and its curve is closer to the real image than other generation methods.

[0087] Quantitative analysis:

[0088] The following table summarizes the quantitative comparison results of this method with CycleGAN, CUT, DCL and Pix2pix in terms of NRMSE, PSNR and SSIM. For SDACL, the mean ± standard deviation of NRMSE, PSNR and SSIM of the generated T1CE images are 0.307 ± 0.12, 23.337 ± 3.21 and 0.881 ± 0.05, respectively. The gaps with the quantitative evaluation results of CycleGAN are: -0.054, 1.476 and 0.047, the gaps with the quantitative evaluation results of CUT are: -0.072, 2.38 and 0.04, the gaps with the quantitative evaluation results of DCL are: -0.021, 0.85 and 0.016, and the gaps with the quantitative results of Pix2pix are: 0.051, 0.579 and 0.032. SDACL outperforms the other four methods in three indicators;

[0089]

[0090]

[0091] Subjective evaluation:

[0092] Radiologists scored the images as Fig. 9 As shown in the figure, the scores of images generated by CUT are all 1 or 2, and most of the images are scored 1, which is the worst image quality. Most of the images generated by CycleGAN are scored 2, and a few are scored 3. Most of the images generated by DCL are scored 3. The quality of most images is above the average standard. The number of 5-point images is second only to SDACL. The scores of images generated by Pix2pix are more evenly distributed, most of which are 3 or 4. In SDACL, most of the generated images are scored 4. In all experiments, the number of 5-point images is the largest, and the score distribution is closest to the real image. The average score of SDACL is 4.1±0.74, which is much better than CUT (1.2±0.42), CycleGAN (1.9±0.57), DCL (3.4±0.97) and Pix2pix (3.2±1.03).

[0093] In the ablation experiment, the proposed final model was compared with three downgraded models. The quantitative evaluation results are shown in the following table. First, after replacing the dilated convolution with the traditional convolution, the NRMSE, PSNR and SSIM of model A become 0.321, 22.672 and 0.87. Secondly, after excluding the attention mechanism module, the three indicators of model B become 0.332, 22.929 and 0.871. Finally, after removing the SSIM loss, the three indicators of model C become 0.34, 22.786 and 0.863. The results show that each component in the model can improve the quality of the generated image to a certain extent.

[0094]

[0095]

[0096] The results of different sequence inputs are shown in the following table. When only T1 input is used, satisfactory performance is obtained. The results of the three indicators are 0.325, 22.796, and 0.872, respectively. The results are best in single sequence input. However, the addition of T2 and Flair introduces additional information, which can further improve the quality of generated images. When T1 and T2 sequences are used as input, the results of the three indicators are 0.321, 23.115, and 0.873, respectively. When T1 and Flair sequences are used as input, the results of the three indicators are 0.328, 22.014, and 0.852, respectively. Compared with the Flair sequence, the T2 sequence contains more information. It can be seen from the results that the results of multimodal input are significantly better than those of single modal input. This is because the attention module and dilated convolution of the SDACL model enhance the expression ability of the model and improve the quality of generated images by fusing the features of multi-sequence MRI images.

[0097]

[0098] In addition, the choice of patch size plays a key role in contrastive learning. Smaller patch sizes may lead to increased fragmentation of semantic content within embedded features, which may affect the model's ability to establish consistency between different regions and lead to incorrect contrastive representation learning. On the other hand, larger patch sizes that greatly exceed the size of the tumor region within the image may cover both pathological and normal regions, resulting in confusing content. These settings, in turn, have a negative impact on model performance. After careful experiments and analysis, the model selects appropriate patch sizes to balance information capture in different regions, namely 9×9, 15×15, 35×35, and 99×99, which ensures that the model can accurately capture the features of the tumor region while maintaining the overall consistency of the generated image.

[0099] It should be noted that the structure described in the present invention can be implemented in a variety of different forms and is not limited to the described embodiments. Any equivalent transformations made by ordinary technicians in this field using the contents of the present invention description and drawings, or directly or indirectly applied to other related technical fields, such as the loading and unloading of other items, are included in the protection scope of the present invention.

Claims

1. A method for generating magnetic resonance gadolinium contrast-enhanced images based on contrast learning, characterized in that: The SDACL model includes a domain X representing a multimodal MRI channel superposition image and a domain Y representing a T1CE image; Generators G and F are both expansion generators based on the attention mechanism module CBAM and dilated convolution introduced in the ResNet architecture. They are used to take T1CE images as input and generate enhanced images through the CBAM module; The discriminator network uses the Markov discriminator PatchGAN as the discriminator network, which includes the first four convolutional layers for obtaining image features and the last convolutional layer for further processing the extracted features to provide suitable input for the fully connected layer; It also includes patch-based multi-layer contrastive learning of the SDACL model and the adjustment of the designed loss function.

2. The method for generating magnetic resonance gadolinium contrast-enhanced images based on contrast learning as claimed in claim 1, characterized in that: The dilation generator includes 2 downsampling blocks, 9 residual blocks and 2 upsampling blocks. Each upsampling block and downsampling block follows a two-step convolution / deconvolution, normalization, and ReLU structure. Each residual block includes dilated convolution, normalization, ReLU, dilated convolution, normalization, CBAM, and residual connection. It also includes 6 convolutional layers, and the convolution kernel sizes are 7×7, 3×3, 3×3, 3×3, 3×3 and 7×7, and the number of filters are 64, 128, 256, 128, 64 and 1 respectively.

3. The method for generating magnetic resonance gadolinium contrast-enhanced images based on contrast learning as claimed in claim 1, characterized in that: The attention mechanism module CBAM includes a channel attention module CAM and a spatial attention module SAM. CAM is used to adaptively select important channel features and filter out unimportant channel information, and SAM is used to focus on the most useful spatial part of the image.

4. The method for generating magnetic resonance gadolinium contrast-enhanced images based on contrast learning as claimed in claim 1, characterized in that: The Markov discriminator PatchGAN includes 5 convolutional layers. The first 4 convolutional layers are used to obtain image features, including four 4×4 convolutional layers, instance normalization layers and LeakyReLU activation functions. The slope is set to 0.

2. The slope of the activation function is set to 0.

2. The step size of each convolutional layer is 2, and the step size of the last layer is 1. The output channels are 64, 128, 256 and 512 respectively. After the last convolutional layer, the discriminator uses a fully connected layer to output a one-dimensional predicted feature map to determine the degree of match between the input data and the given conditions.

5. The method for generating magnetic resonance gadolinium contrast-enhanced images based on contrast learning as claimed in claim 1, characterized in that: The SDACL model uses PatchNCE loss to match the corresponding patches of the input image and the output image. Each feature represents a patch in the image. The corresponding features in the selected layer are used as positive samples, and other features are used as negative samples. The query patch, positive samples and negative samples are mapped to the K-dimensional vector space and expressed as: v、v + ∈R K and v - ∈R N×K ; Among them, v - ∈R N×K represents the nth negative sample patch; Then, the vector is normalized using L2 normalization, and then an (N+1)-way classification problem is established to calculate the probability of selecting a positive sample instead of a negative sample, where the distance between the sample and other positive and negative samples is τ. The cross entropy loss function is used to indicate that the probability of a positive sample being selected is greater than the probability of a negative sample. The function expression is: Patch-based multi-layer contrastive learning maps the features of the input image and the generated image into the same embedding space, and the NCE loss forces the features of the generated T1CE to remain close to the features of patches in multiple plain scan sequences located at the same position, while staying away from other patches.

6. The method for generating magnetic resonance gadolinium contrast-enhanced images based on contrast learning as claimed in claim 1, characterized in that: The loss functions designed for the SDACL model include the adversarial loss GANLoss, which is used to train the generator and the discriminator so that the images generated by the generator are closer and closer to the real images, the patch-level contrast loss PatchNCELoss, which is used to calculate the similarity between the generated images and the real images at the patch level, and the structural similarity loss SSIMLoss, which is used to measure the structural similarity between the generated images and the real images and helps to maintain the details and texture of the images.

Images