A cross-modality brain gadolinium contrast-enhanced MRI image synthesis method and system
Patent Information
- Application Number
- CN202411384586.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2026-10-09
AI Technical Summary
然而，传统的局部感知方法通常共享共同的底层特征，并针对多个可能存在冲突的目标，导致特征冲突
[0067]本发明的有益效果为：本发明提供了一种创新的脑钆对比增强MRI图像合成方法和系统，通过减少对患者的钆基对比剂依赖，显著降低了与对比剂使用相关的健康风险，如过敏反应和副作用，从而提升了影像检查的安全性。该方法通过先进的跨模态合成技术生成高质量的钆增强MRI图像，确保病灶区域的细节清晰，提高了影像的诊断价值和准确性。此外，减少对比剂使用还降低了医疗成本，同时提高了医疗资源的可及性，特别是在对比剂供应有限的地区。通过个性化图像生成，该技术支持了更精确的疾病诊断和治疗规划，推动了个性化医疗的发展。最终，本发明不仅在临床应用中表现出色，还为神经科学研究和医学教学提供了可靠的数据支持和真实的教学资源，全面提升了影像检查的安全性和效果。
Smart Images
Figure CN122888030A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of magnetic resonance imaging, and more particularly to a method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images of the brain. Background Technology
[0002] Magnetic resonance imaging (MRI) is a widely used technique for detecting brain diseases. It uses strong magnetic fields and radiofrequency waves to generate detailed images of internal structures. Gadolinium-based contrast agents are commonly used in brain MRI scans because they significantly improve image contrast and clarity, helping doctors diagnose brain lesions such as tumors, inflammation, and vascular malformations more accurately. However, the use of gadolinium-based contrast agents is not without risks; they may cause potential side effects, including allergic reactions and brain deposits. Furthermore, in patients with renal insufficiency, the use of gadolinium-based contrast agents may increase the risk of developing renal systemic fibrosis. Therefore, there is an urgent need to find safe alternatives.
[0003] In recent years, the development of cross-modal medical image synthesis methods has provided a promising alternative to the use of gadolinium-based contrast agents. These techniques focus on using deep learning models to map the relationship between source and target modal images. Multimodal image fusion utilizes complementary information between different modalities to generate high-quality globally synthesized images. However, synthesizing high-quality brain MRI images faces two major challenges: first, brain lesions, such as cerebral hemorrhage and gliomas, vary greatly in size, shape, and location; second, some lesions, although occupying only a small portion of brain MRI images and associated with high background contrast, are difficult to synthesize. For example, the edema surrounding a glioma appears as a wreath in gadolinium-enhanced MRI images; although small, it has significant clinical implications in tumor diagnosis.
[0004] To address this challenge, some studies have introduced multimodal models that incorporate local perception tasks. These models aim to provide additional supervisory signals specifically for local lesion regions, employing auxiliary tasks such as edge generation or lesion segmentation. This targeted supplementary signal aims to enhance attention to lesion regions during image generation. However, traditional local perception methods often share common low-level features and target multiple potentially conflicting objectives, leading to feature conflicts. Furthermore, there is often a lack of sufficient interaction between local perception and global learning, which can result in discrepancies between lesion edges or semantic information captured by local tasks and the globally synthesized image. Summary of the Invention
[0005] To address the potential patient risks associated with the use of gadolinium-based contrast agents in MRI scans and the limitations of existing cross-modal synthesis techniques in synthesizing high-quality gadolinium-enhanced MRI images of lesions, this invention proposes a cross-modal method and system for synthesizing gadolinium-enhanced brain MRI images. This provides a safe alternative to gadolinium contrast agents and can be widely applied in medical diagnosis, treatment monitoring, and neuroscience research, improving the safety and effectiveness of imaging examinations.
[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows: a method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images, characterized in that the steps include:
[0007] S1: Acquire T1, T2, Flair, and T1 ce modal images of brain MRI scans from different subjects;
[0008] S2: Delineate brain disease lesions on the T1 ce modality image to obtain lesion segmentation map, and use the Sobel algorithm to obtain texture edge map of brain disease lesions;
[0009] S3: Constructing a cross-modal brain gadolinium contrast-enhanced MRI image synthesis network;
[0010] S4: The lesion segmentation map and edge map are used as labels for the lesion segmentation and edge generation tasks in the local perception module, respectively. The cross-modal brain gadolinium contrast-enhanced MRI image synthesis network is trained using T1, T2, Flair and T1 ce modal images. The quality of the generated images is comprehensively supervised from multiple targets using the total loss function.
[0011] S5: The network trained in inference step S4 generates T1 ce modal images from T1, T2, and Flair modal images of brain MRI scans, thus obtaining a method for synthesizing cross-modal gadolinium contrast-enhanced MRI images of the brain.
[0012] The method for obtaining T1, T2, FLAIR, and T1 ce modal images of brain MRI scans from different subjects, as described in step S1, is as follows:
[0013] Subjects were placed inside an MRI scanner, and their brains were scanned using axial T1, T2, FLAIR, and T1 ce sequences. The resulting T1, T2, FLAIR, and T1 ce modal images were used as the scan images. The slice thickness for the T1 and T1 ce sequences was 5.5–6.5 mm, the repetition time was 220–1970 ms, and the echo time was 2.46–28.60 ms. The slice thickness for the T2 and FLAIR sequences was 4–6.5 mm, the repetition time was 3000–5630 ms, and the echo time was 90–118 ms. All scan images were rescaled to 256 × 256 pixels and normalized to the range [-1, 1]. The T1, T2, and FLAIR modal images were used as model inputs, and the T1 ce modal image was used as the label for the model's generated results. The scan images were randomly divided into training and test sets at a 4:1 ratio.
[0014] The method for obtaining the texture edge map of brain disease lesions using the Sobel algorithm in step S2 is as follows:
[0015] The disease lesion region in the acquired head T1 ce modality images was delineated using 3D slicer software, and the lesion segmentation map Y was obtained. s As a label for the lesion segmentation task in the local perception module;
[0016] Texture features were extracted from the head T1 ce modality image using the Sobel algorithm:
[0017]
[0018] in, and Let Y and Y represent the Sobel gradients of the T1 ce modality image along the x-axis and y-axis, respectively. e It is a synthesized edge texture map, which will be Y e As a label for the edge generation task in the local perception module.
[0019] The cross-modal gadolinium contrast-enhanced MRI image synthesis network described in step S3 includes a multimodal coding module, a feature decoupling module, a local perception module, and a global fusion module. The multimodal coding module and the local perception module are both connected to the feature decoupling module, and the local perception module and the multimodal coding module are both connected to the global fusion module.
[0020] The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 4, characterized in that the multimodal coding module takes T1 modality images, T2 modality images, and FLAIR modality images as input, and each modality is represented by the symbol X. iLet X be an input modality, where i = 1, 2, 3, and i represents the T1, T2, and Flair sequences; X represents each input modality X. i From the corresponding encoder branch The processing involves three downsampling convolutional layers of the encoder, where j = 1, 2, 3, and j represents the three layers of the encoder. The encoder includes a ReLU activation function and layer instance normalization. Multimodal deep features F are obtained by connecting the deep features of each branch. d ;
[0021]
[0022] The feature decoupling module includes a parallel CNN module and a Mamba module, using multimodal deep features F d As input;
[0023] The CNN module consists of six sequentially connected residual convolutional layers (ResCNN), and the global edge texture features F are obtained through de-entanglement. h , of which F h =ResCNN(F d );
[0024] The Mamba module includes two parallel branches. The Mamba module is used to perform layer normalization on the input to obtain normalized data, and the normalized data is processed by the two branches respectively.
[0025] The first branch consists of a multilayer perceptron, a convolutional neural network, a SiLU activation function, and a state-space model connected in sequence; the first branch is expressed by the formula:
[0026]
[0027] The second branch is the gate control mechanism, which includes a multilayer perceptron and a SiLU activation function connected in sequence. The second branch is expressed by the following formula:
[0028]
[0029] The final output F, representing the semantic features of the global lesions, is calculated by combining the outputs of the two branches. l :
[0030]
[0031] The local perception module includes an edge generation task E branch and a lesion segmentation task S branch. The edge generation task E branch includes three sequentially connected upsampled convolutions, using Sigmoid as the activation function; the lesion segmentation task S branch includes three sequentially connected upsampled convolutions, using Softmax as the activation function.
[0032] edge texture map Ye As labels for the edge generation task in the local perception module, the expressions for the edge generation task E branch and the lesion segmentation task S branch are:
[0033]
[0034] Among them, E j S represents the branch encoder of edge generation task E. j The branch encoder represents the lesion segmentation task S, j = 1, 2, 3; This represents the texture edge map of the target modality output by the local sensing module. This represents the lesion segmentation map of the target modality output by the local sensing module.
[0035] The global fusion module includes a full fusion module and an attention mechanism module connected in sequence. The global fusion module passes the encoder downsampling layer features to the full fusion module through skip connections, and then uses the attention mechanism to achieve global thinning of the synthesized image.
[0036] The method for global thinning of synthesized images using an attention mechanism is as follows:
[0037] First, perform average pooling F. g To obtain the global information σ for each channel, a weighting factor σ′ is obtained by using the sigmoid activation function to learn the global information σ for each channel. The weighting factor σ′ represents the feature map F. g The correlation of the mapping to the target modal image;
[0038]
[0039] σ=AvgPool(F g ),
[0040] σ′=Sigmoid(σ),
[0041] in, High-frequency edge features The semantic features of low-frequency lesions These are the underlying detailed features that have undergone jump connections. Features of the fusion result from the previous layer;
[0042] The input to the first-layer global fusion module GFM includes high-frequency edge features, low-frequency lesion semantic features, and low-level detail features after jump connections. The expression for the global fusion module GFM is:
[0043]
[0044] Use weighting factors on feature map F gReweighting to highlight features associated with the target modality map:
[0045]
[0046] The reweighted feature map F g ′ and the original feature map F g Add them together to minimize the feature F. g The information loss is then accounted for, and finally, the transposed convolution is upsampled again to obtain the potential target modality features. The expression is:
[0047]
[0048] Synthetic modality images are obtained by applying the Tanh activation function:
[0049]
[0050] The method described in step S4, which utilizes the total loss function to comprehensively supervise the quality of generated images from multiple targets, is as follows:
[0051] The total loss function includes reconstruction loss, feature decoupling loss, lesion segmentation loss, edge generation loss, and discrimination loss. First, the reconstruction loss L of the T1 ce modality image is calculated. rec :
[0052] L rec =L1+α·L SSIM ;
[0053]
[0054] Among them, L1 loss emphasizes pixel-level differences, L SSIM The loss considers the structural and perceptual similarity of the images, with α as a weighting factor, x and y representing the labels of the synthesized T1 ce modality image and the real T1 ce modality image, respectively, and μ x and μ y σ represents the mean brightness of the synthesized T1 ce modal image and the real T1 ce modal image, respectively. x and σ y Let σ represent the luminance variance of the synthesized T1 ce modal image and the real T1 ce modal image, respectively. xy The brightness covariance of the synthesized T1 ce modal image and the real T1 ce modal image is represented, which measures the relationship between their brightness changes. C1 and C2 are two small constants used to stabilize the division operation.
[0055] Based on prior knowledge, a feature-based deentanglement loss function L is designed. de ;
[0056]
[0057] L de =L cor +L uncor ;
[0058] Where cos(.) represents the average cosine similarity, L cor L represents the correlation loss between high-frequency edge features in multimodal analysis. uncor L represents the loss of uncorrelation between low-frequency lesion semantic features in multimodal analysis. de It is the sum of the two, the index set S = {(0,1), (1,2), (0,2)}, the index set represents the pairwise correlation loss between multiple modalities of high-frequency edge features and the pairwise non-correlation loss between low-frequency lesion semantic features;
[0059] Lesion segmentation loss L sem Edge generation loss L edge They are as follows:
[0060]
[0061] Discriminant loss L based on WassersteinGAN D for:
[0062]
[0063] Among them, f w Let x represent the discriminator;
[0064] The total loss function is defined as follows:
[0065] L = L rec +β·L de +γ·L sem +δ·L edge +L D ;
[0066] Where β, γ and δ are all weighting factors.
[0067] The beneficial effects of this invention are as follows: This invention provides an innovative method and system for synthesizing gadolinium-contrast-enhanced MRI images of the brain. By reducing patients' dependence on gadolinium-based contrast agents, it significantly reduces health risks associated with contrast agent use, such as allergic reactions and side effects, thereby improving the safety of imaging examinations. This method generates high-quality gadolinium-enhanced MRI images through advanced cross-modal synthesis technology, ensuring clear details in lesion areas and improving the diagnostic value and accuracy of the images. Furthermore, reducing contrast agent use lowers medical costs while improving the accessibility of medical resources, especially in areas with limited contrast agent supplies. Through personalized image generation, this technology supports more precise disease diagnosis and treatment planning, promoting the development of personalized medicine. Ultimately, this invention not only performs excellently in clinical applications but also provides reliable data support and authentic teaching resources for neuroscience research and medical education, comprehensively improving the safety and effectiveness of imaging examinations. Attached Figure Description
[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0069] Figure 1 This is a schematic flowchart of the cross-modal gadolinium contrast-enhanced MRI image synthesis method according to an embodiment of the present invention.
[0070] Figure 2 This is a structural framework diagram of the cross-modal brain gadolinium contrast-enhanced MRI image synthesis network according to an embodiment of the present invention.
[0071] Figure 3 This is a data processing flowchart of the feature decoupling module in an embodiment of the present invention.
[0072] Figure 4 This is a data processing flowchart of the global fusion module in an embodiment of the present invention. Detailed Implementation
[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0074] like Figure 1 As shown, a method for synthesizing cross-modal gadolinium contrast-enhanced MRI images of the brain includes the following steps:
[0075] S1: Acquire T1, T2, Flair, and T1 ce modal images of the brains of different subjects using Magnetic Resonance Imaging (MRI).
[0076] Specifically, the subject was first placed in an MRI scanner to perform axial T1, T2, Flair (fluidized inversion recovery) and T1 ce sequences to scan the subject's brain, obtaining T1 modality images, T2 modality images, Flair modality images, and T1 ce modality images as scan images. The slice thickness for the T1 and T1 ce sequences was 5.5–6.5 mm, the repetition time was 220–1970 ms, and the echo time was 2.46–28.60 ms. The slice thickness for the T2 and Flair sequences was 4–6.5 mm, the repetition time was 3000–5630 ms, and the echo time was 90–118 ms. All scan images were rescaled to 256 × 256 pixels and the values were normalized to the range [-1, 1]. The T1, T2, and FLAIR modal images from the scanned images were used as model inputs, while the T1 ce modal image served as the label for the model's generated results. The data was randomly divided into training and test sets in a 4:1 ratio.
[0077] S2: Delineate brain disease lesions on the T1 ce modality image to obtain a lesion segmentation map, and simultaneously use the Sobel algorithm to obtain the texture edge map of the brain disease lesions.
[0078] Specifically, 3D slicer software was used to delineate the disease lesion regions in the acquired head T1 ce modality images, and the lesion segmentation map Y was saved. s As a label for the lesion segmentation task in the local perception module, the Sobel algorithm is used to extract texture features from the head T1ce modality image: in, and Let Y and Y represent the Sobel gradients of the T1 ce modality image along the x-axis and y-axis, respectively. e It is a synthesized edge texture map, which provides edge information of the image and serves as a label for the edge generation task in the local perception module.
[0079] S3: Constructing a cross-modal gadolinium contrast-enhanced MRI image synthesis network can reduce the potential harm caused by the use of gadolinium-enhanced contrast agents, save medical costs, improve image synthesis quality, enhance feature expression capabilities, optimize the fusion of local and global information, improve the efficiency of multimodal data fusion, and provide more accurate information in medical diagnosis, treatment monitoring, and neuroscience research.
[0080] Specifically, such as Figure 2 As shown, the cross-modal brain gadolinium contrast-enhanced MRI image synthesis network includes a multimodal encoding module (MEM), a feature decoupling module (FDM), a local perception module (LPM), and a global fusion module (GFM). The multimodal encoding module, the local perception module, and the global fusion module are all connected to the feature decoupling module, and the local perception module and the multimodal encoding module are all connected to the global fusion module.
[0081] The multimodal coding module is designed to accept multimodal input from non-contrast-enhanced MRI. Each modality is represented by the symbol X. i Let i = 1, 2, 3 represent T1, T2, and Flair sequences, respectively. Each input mode X... i From its dedicated encoder branch The processing involves three downsampling convolutional layers of the encoder, where j = 1, 2, and 3 represent the three layers. Furthermore, the encoder incorporates ReLU activation and layer instance normalization to stabilize training and extract depth feature representations for each MRI modality. Finally, the multimodal depth features F are obtained by concatenating the depth features of each branch. d .
[0082]
[0083] like Figure 3 As shown, the feature decoupling module adopts a parallel structure using CNN-Mamba, including a CNN module and a Mamba module. The feature decoupling module decouples multimodal deep features F d As input.
[0084] In image processing, the local receptive fields of shallow CNNs enable them to capture high-frequency edge features in images more effectively.
[0085] The CNN module is primarily used for extracting local edge features. It consists of six sequentially connected ResCNN convolutional layers, each using the ReLU activation function. Each layer has a 3×3 kernel size and a stride of 1. The global edge texture features F are obtained through de-entanglement. h Represented as F h =ResCNN(F d ).
[0086] In existing technologies, the Transformer model has a relatively mature ability to extract global semantic features. However, in practical applications, the Transformer model also faces many limitations due to the secondary computational complexity associated with its self-attention mechanism. To address this issue, this application utilizes the Mamba architecture to introduce an innovative method by integrating a Selective State Space Model (SSM) module, thereby improving the efficiency of extracting global low-frequency lesion semantic features.
[0087] First, the input is layer-normalized using the Mamba module, and then the normalized data is processed through two different branches.
[0088] The first branch consists of a multilayer perceptron, a convolutional neural network (CNN), a SiLU activation function, and a state-space model, connected sequentially. Normalized data first encounters a multilayer perceptron (MLP) to resize the features. After resizing, the CNN effectively analyzes local patterns and spatial relationships within the normalized data. The CNN extracts local features from the data through its convolutional layers, capturing local patterns (such as edges and textures), and progressively builds higher-level feature representations through multiple convolutional layers. This allows the CNN to learn complex spatial structures and patterns in the data, thereby improving model performance and prediction accuracy. Subsequently, the SiLU activation function is applied to introduce a nonlinear transformation to the normalized data, which helps improve feature selection. Specifically, the SiLU activation function applies a nonlinear transformation to the normalized data, enabling the data to capture more complex feature relationships, thus effectively improving the model's expressive power and performance. This nonlinear transformation allows the model to better adapt to complex patterns in the data and play a greater role in feature selection. The final component of the first branch is the state-space model (SSM), whose task is to efficiently extract global semantic features. The first branch is expressed by the formula:
[0089]
[0090] The second branch is a gating mechanism, consisting of a multilayer perceptron and a SiLU activation function connected in sequence. The second branch processes the input through the MLP and SiLU activation function to generate a gating factor. This factor is used to adaptively select features more important for lesion segmentation from the features extracted from the first branch. The second branch is expressed by the following formula:
[0091]
[0092] The final output F, representing the semantic features of the global lesions, is computed by combining the outputs from these two branches. l:
[0093]
[0094] The local perception module consists of an edge generation task E and a lesion segmentation task S. The local perception module obtains high-frequency edge features F from the texture edges of the target modality T1ce and the signal supervision of the lesion segmentation map. h and low-frequency lesion semantic features F l Local potential target features in the T1ce lesion region of the learning target modality.
[0095] Both the edge generation and lesion segmentation task branches consist of three upsampled convolutions. The first two convolutions use 3×3 kernels, and the last convolution uses a 7×7 kernel. The difference between the two branches is that the last activation function used for edge generation is sigmoid, while lesion segmentation uses softmax. The texture map Y... e As a label for the edge generation task in the local perception module.
[0096]
[0097] Among them, F j S represents the branch encoder of edge generation task E. j Let j = 1, 2, 3 represent the branch encoder for the lesion segmentation task S. The texture edge map representing the target modality output by the model. This represents the lesion segmentation map of the target mode output by the model.
[0098] like Figure 4 As shown, the global fusion module is mainly used to refine the global latent target modal features. The global fusion module is based on the local latent target features output by the local perception module. Specifically, firstly, through skip connections, the encoder passes the sampling layer features to the global fusion module to supplement the detailed features lost due to deentanglement. Then, an attention mechanism is used to gradually fuse the local perception features, detailed features, and the global latent target features from the previous layer, ultimately achieving global refinement of the synthesized image.
[0099] Specifically, the inputs to this module are the edge synthesis task branch E. j Features, lesion segmentation task branches S j Features, underlying details passed through jump connections and global potential target features obtained from the previous layer global fusion module Note the following formula for the mechanism: First, perform average pooling to obtain global information σ for each channel. Then, use the sigmoid activation function to learn the global information σ for each channel to obtain the weighting factor σ′. The weighting factor σ′ represents the feature map F. gThe correlation of the mapping to the target modal image.
[0100]
[0101] σ=AvgPool(F g ),
[0102] σ′=Sigmoid(σ),
[0103] Then, the feature map F is weighted by a weighting factor. g Reweighting to highlight features associated with the target modality map.
[0104]
[0105] The reweighted feature map F g ′ and the original feature map F g Add them together to minimize the feature F. g The information loss is considered. Finally, the transposed convolution is upsampled again to obtain the potential target modality features.
[0106]
[0107] In summary, the Global Fusion Module (GFM) uses an attention mechanism to integrate high-frequency edge features from each layer. low-frequency lesion semantic features After jump connection The underlying details and the previous layer The fusion results are progressively fused to obtain each layer. The global refinement of potential target modal features. It is worth mentioning that the first-layer global fusion module GFM has no input from the previous layer.
[0108]
[0109] Synthetic modality images are obtained by applying the Tanh activation function:
[0110]
[0111] The loss function, including reconstruction loss, feature decoupling loss, lesion segmentation loss, edge generation loss, and discrimination loss, can comprehensively supervise the quality of the generated image from multiple objectives. This multi-objective supervision method ensures optimization of the image in terms of overall quality, feature representation, lesion segmentation, edge sharpness, and realism, thereby improving the overall performance and application effect of the generated image.
[0112] Calculate the reconstruction loss L of the T1 ce modality image. rec :
[0113] L rec =L1+α·L SSIM ;
[0114]
[0115] Among them, L1 loss emphasizes pixel-level differences, L SSIM The loss considers the structural and perceptual similarity of the images, with α as a weighting factor, x and y representing the labels of the synthesized T1 ce modality image and the real T1 ce modality image, respectively, and μ x and μ y σ represents the mean brightness of the synthesized T1 ce modal image and the real T1 ce modal image, respectively. x and σ y Let σ represent the luminance variance of the synthesized T1 ce modal image and the real T1 ce modal image, respectively. xy The brightness covariance of the synthesized T1 ce modal image and the real T1 ce modal image is represented, which measures the relationship between their brightness changes. C1 and C2 are two small constants used to stabilize the division operation and prevent the denominator from being zero.
[0116] To better encourage multimodal feature deentanglement, a feature deentanglement loss function L was designed based on prior knowledge. de This means that the correlation between high-frequency edge features and the correlation between low-frequency lesion semantic features are low among MRI multimodal images, while the correlation between low-frequency lesion semantic features is high, where cos(.) represents the average cosine similarity, and L... cor L represents the correlation loss between high-frequency edge features in multimodal analysis. uncor L represents the loss of uncorrelation between low-frequency lesion semantic features in multimodal analysis. de It is the sum of the two.
[0117]
[0118] L de =L cor +L uncor ;
[0119] The index set is defined as S = {(0, 1), (1, 2), (0, 2)}, which represents the pairwise correlation loss between multiple modalities of high-frequency edge features and the pairwise non-correlation loss between low-frequency lesion semantic features.
[0120] Lesion segmentation loss L sem Edge generation loss L edge They are as follows:
[0121]
[0122] Discriminant loss L based on WassersteinGAN D for:
[0123]
[0124] Among them, f w Let x represent the discriminator, and let x represent the input of the discriminator.
[0125] The total loss function is defined as follows:
[0126] L = L rec +β·L de +γ·L sem +δ·L edge +L D ;
[0127] Where β, γ and δ are all weighting factors.
[0128] S4: The cross-modal gadolinium contrast-enhanced MRI image synthesis network constructed in step S3 is trained. The T1, T2, and FLAIR modal images obtained in step 1 are used as model inputs, and the T1ce modal images are used as output labels to supervise the generated pseudo-T1ce modal images. The lesion segmentation map and edge map generated in step S2 are used as labels for the lesion segmentation and edge generation tasks in the local perception module, respectively.
[0129] Further, step 4 trains a cross-modal gadolinium contrast-enhanced MRI image synthesis network. The T1, T2, and Flair modal images obtained in step c1 are used as input, and the T1ce modal image is used as the label for the synthesized gadolinium-enhanced MRI image. The lesion segmentation map and edge map generated in step 2 are used as labels for lesion segmentation and edge generation tasks in the local perception module, respectively. Specifically, the experiment was conducted in a PyTorch environment using a 1×NVIDIA GeForce RTX 4090 GPU. The RMSProp optimizer was used, with a batch size of 4. The number of channels in the multimodal input image was 1, and both the length and width were cropped by 256. The optimizer used RMSprop with an initial learning rate of 1e-4, trained for 300 epochs, and the learning rate decayed starting from 200 epochs. Pixel values were normalized to [-1, 1] when the equalization parameters β = 0.1, γ = 0.1, and δ = 0.1. The T1, T2, and Flair modal images obtained in step 1 are used as input, and the T1ce modal image is used as the label for the synthesized gadolinium-enhanced MRI brain image. The lesion segmentation map and edge map generated in step 2 are used as the labels for the lesion segmentation and edge generation tasks in the local perception module, respectively.
[0130] S5: The network trained in the inference step S4 generates T1 ce modality images from T1, T2, and Flair modality images of brain MRI scans, providing a safer alternative to gadolinium contrast agents.
[0131] Using the network trained in step S4, pseudo-T1ce modal images are generated based on T1, T2, and FLAIR modal images from brain MRI scans, providing a safer alternative to gadolinium contrast agents. The process is as follows: T1, T2, and FLAIR modal images from the test set obtained from brain MRI scans are input into the model trained in step 4, and the T1ce target modal image corresponding to the patient is inferred as an alternative to gadolinium contrast agent-enhanced scanning.
[0132] This invention proposes an innovative method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images of the brain, specifically designed to generate high-quality gadolinium-enhanced brain MRI images with lesion regions. The Feature Separation Module (FDM) first separates multimodal depth features into high-frequency edge features and low-frequency lesion semantic features, effectively resolving feature conflicts arising from different learning objectives in local perception tasks. Subsequently, the Local Perception Module (LPM) processes these separated features, extracting lesion-specific perceptual features through tasks such as edge generation and lesion segmentation. To reduce potential information loss due to separation, basic detail features are reintegrated through jump connections. Finally, the Global Fusion Module (GFM) progressively fuses and enhances these local perceptual and detail features through an attention mechanism, meticulously crafting a global target modality representation. This invention provides a safe and high-quality alternative for gadolinium contrast-enhanced brain MRI scanning, which can be widely applied in medical diagnosis, treatment monitoring, and neuroscience research, improving the safety and effectiveness of imaging examinations.
[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images of the brain, characterized in that the steps include... include: S1: Acquire T1, T2, Flair, and T1 ce modal images of brain MRI scans from different subjects; S2: Delineate brain disease lesions on the T1 ce modality image to obtain lesion segmentation map, and use the Sobel algorithm to obtain texture edge map of brain disease lesions; S3: Constructing a cross-modal brain gadolinium contrast-enhanced MRI image synthesis network; S4: The lesion segmentation map and edge map are used as labels for the lesion segmentation and edge generation tasks in the local perception module, respectively. The cross-modal brain gadolinium contrast-enhanced MRI image synthesis network is trained using T1, T2, Flair and T1 ce modal images. The quality of the generated images is comprehensively supervised from multiple targets using the total loss function. S5: The network trained in inference step S4 generates T1 ce modal images from T1, T2, and Flair modal images of brain MRI scans, thus obtaining a method for synthesizing cross-modal gadolinium contrast-enhanced MRI images of the brain.
2. The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 1, characterized in that, The method for obtaining T1, T2, FLAIR, and T1 ce modal images of brain MRI scans from different subjects, as described in step S1, is as follows: Subjects were placed inside an MRI scanner, and their brains were scanned using axial T1, T2, FLAIR, and T1 ce sequences. The resulting T1, T2, FLAIR, and T1 ce modal images were used as the scan images. The slice thickness for the T1 and T1 ce sequences was 5.5–6.5 mm, the repetition time was 220–1970 ms, and the echo time was 2.46–28.60 ms. The slice thickness for the T2 and FLAIR sequences was 4–6.5 mm, the repetition time was 3000–5630 ms, and the echo time was 90–118 ms. All scan images were rescaled to 256 × 256 pixels and normalized to the range [-1, 1]. The T1, T2, and FLAIR modal images were used as model inputs, and the T1 ce modal image was used as the label for the model's generated results. The scan images were randomly divided into training and test sets at a 4:1 ratio.
3. The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 2, characterized in that, The method for obtaining the texture edge map of brain disease lesions using the Sobel algorithm in step S2 is as follows: The disease lesion region in the acquired head T1 ce modality images was delineated using 3D slicer software, and the lesion segmentation map Y was obtained. s As a label for the lesion segmentation task in the local perception module; Texture extraction from head T1 ce modality image using the Sobel algorithm feature: in, and Let Y and Y represent the Sobel gradients of the T1 ce modality image along the x-axis and y-axis, respectively. e It is a synthesized edge texture map, which will be Y e As a label for the edge generation task in the local perception module.
4. The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 3, characterized in that, The cross-modal gadolinium contrast-enhanced MRI image synthesis network described in step S3 includes a multimodal coding module, a feature decoupling module, a local perception module, and a global fusion module. The multimodal coding module and the local perception module are both connected to the feature decoupling module, and the local perception module and the multimodal coding module are both connected to the global fusion module.
5. The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 4, characterized in that, The multimodal coding module takes T1 modal image, T2 modal image and Flair modal image as input, and each modality is represented by the symbol X. i Let X be an input modality, where i = 1, 2, 3, and i represents the T1, T2, and Flair sequences; X represents each input modality X. i From the corresponding encoder branch The processing involves three downsampling convolutional layers of the encoder, where j = 1, 2, 3, and j represents the three layers of the encoder. The encoder includes a ReLU activation function and layer instance normalization. Multimodal deep features F are obtained by connecting the deep features of each branch. d ; 6. The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 4 or 5, characterized in that, The feature decoupling module includes a parallel CNN module and a Mamba module, using multimodal deep features F d As input; The CNN module consists of six sequentially connected ResCNN convolutional layers, which are used to obtain global edge texture features F through de-entanglement. h , of which F h =ResCNN(F d ); The Mamba module includes two parallel branches. The Mamba module is used to perform layer normalization on the input to obtain normalized data, and the normalized data is processed by the two branches respectively. The first branch consists of a multilayer perceptron, a convolutional neural network, a SiLU activation function, and a state-space model connected in sequence. The first branch is expressed by the formula: The second branch is the gate control mechanism, which includes a multilayer perceptron and a SiLU activation function connected in sequence. The second branch is expressed by the following formula: The final output F, representing the semantic features of the global lesions, is calculated by combining the outputs of the two branches. l :
7. The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 6, characterized in that, The local perception module includes an edge generation task E branch and a lesion segmentation task S branch. The edge generation task E branch includes three sequentially connected upsampled convolutions, using Sigmoid as the activation function; the lesion segmentation task S branch includes three sequentially connected upsampled convolutions, using Softmax as the activation function. edge texture map Y e As labels for the edge generation task in the local perception module, the expressions for the edge generation task E branch and the lesion segmentation task S branch are: Among them, E j S represents the branch encoder of edge generation task E. j The branch encoder represents the lesion segmentation task S, j = 1, 2, 3; This represents the texture edge map of the target modality output by the local sensing module. This represents the lesion segmentation map of the target modality output by the local sensing module.
8. The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 7, characterized in that, The global fusion module includes a full fusion module and an attention mechanism module connected in sequence. The global fusion module passes the encoder downsampling layer features to the full fusion module through skip connections, and then uses the attention mechanism to achieve global thinning of the synthesized image.
9. The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 8, characterized in that, The method for achieving global thinning of synthesized images using an attention mechanism is as follows: First, perform average pooling F. g To obtain the global information σ for each channel, a weighting factor σ′ is obtained by using the sigmoid activation function to learn the global information σ for each channel. The weighting factor σ′ represents the feature map F. g The correlation of the mapping to the target modal image; σ=AvgPool(F g ) σ′=Sigmoid(σ), Among them, F j e For high-frequency edge features, F j s For low-frequency lesion semantic features, F j u These are the underlying detailed features that have undergone jump connections. Features of the fusion result from the previous layer; The input to the first-layer global fusion module GFM includes high-frequency edge features, low-frequency lesion semantic features, and low-level detail features after jump connections. The expression for the global fusion module GFM is: Use weighting factors on feature map F g Reweighting to highlight features associated with the target modality map: The reweighted feature map F g ′ and the original feature map F g Add them together to minimize the feature F. g The information loss is then accounted for, and finally, the transposed convolution is upsampled again to obtain the potential target modality features F. j g The expression is: F j g =ConvT3×3(F g ″); Synthetic modality images are obtained by applying the Tanh activation function:
10. The method and system for synthesizing cross-modal gadolinium contrast-enhanced MRI images according to claim 9, characterized in that, The method described in step S4, which utilizes the total loss function to comprehensively supervise the quality of the generated image from multiple targets, is as follows: The total loss function includes reconstruction loss, feature decoupling loss, lesion segmentation loss, edge generation loss, and discrimination loss. First, the reconstruction loss L of the T1 ce modality image is calculated. rec : L rec =L1+α·L SSIM ; Among them, L1 loss emphasizes pixel-level differences, L SSIM The loss considers the structural and perceptual similarity of the images, with α as a weighting factor, x and y representing the labels of the synthesized T1 ce modality image and the real T1 ce modality image, respectively, and μ x and μ y σ represents the mean brightness of the synthesized T1ce modal image and the real T1ce modal image, respectively. x and σ y Let σ represent the luminance variance of the synthesized T1 ce modal image and the real T1 ce modal image, respectively. xy The brightness covariance of the synthesized T1 ce modal image and the real T1 ce modal image is represented, which measures the relationship between their brightness changes. C1 and C2 are two small constants used to stabilize the division operation. Based on prior knowledge, a feature-based deentanglement loss function L is designed. de ; L de =L cor +L uncor ; Where cos(.) represents the average cosine similarity, L cor L represents the correlation loss between high-frequency edge features in multimodal analysis. uncor L represents the loss of uncorrelation between low-frequency lesion semantic features in multimodal analysis. de It is the sum of the two, the index set S = {(0,1), (1,2), (0,2)}, the index set represents the pairwise correlation loss between multiple modalities of high-frequency edge features and the pairwise non-correlation loss between low-frequency lesion semantic features; Lesion segmentation loss L sem Edge generation loss L edge They are as follows: Discriminant loss L based on WassersteinGAN D for: Among them, f w Let x represent the discriminator; The total loss function is defined as follows: L=L rec +β·L de +γ·L sem +δ·L edge +L D ; Where β, γ and δ are all weighting factors.