Contrast-agent-free DCE-MRI image generation method based on multi-modal feature fusion

Abstract

The invention discloses a contrast-agent-free DCE-MRI image generation method based on multi-modal feature fusion, and belongs to the technical field of medical image processing, and the method comprises the following steps: S1, data preprocessing; s2, multi-source feature extraction; s3, perfusion information attention is enhanced; and S4, image generation. According to the method, a multi-source feature extraction module is adopted, and wavelet transform enhanced feature extraction and apparent dispersion coefficient feature extraction are included, so that the expression ability of different modal information is enriched; an attention module is enhanced through perfusion information, and the recognition effect of a key area is enhanced by using the information complementarity of different modal images; in the generation process, the similarity between the generated image and the target image is optimized through the game process of a generator and a discriminator of the generative adversarial network, and the generator adopts a multi-level convolution block structure to enhance the detail fidelity of the generated image.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for generating contrast agent-free DCE-MRI images based on multimodal feature fusion. Background Art

[0002] In the field of medical imaging, dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is an important imaging technique widely used for tumor detection, disease assessment, and treatment monitoring. However, due to scan time constraints, patient motion, and the complexity of contrast agent distribution, obtaining complete DCE-MRI image sequences is difficult. Therefore, reconstructing or generating complete DCE-MRI sequences has become an important research topic. Currently, existing technologies are mainly divided into physical modeling-based methods and deep learning-based methods.

[0003] Physics-based modeling methods typically use physiological or pharmacokinetic models to describe the dynamic behavior of contrast agents in tissues. In recent years, deep learning methods have made significant progress in medical image generation tasks and have been applied to the reconstruction of DCE-MRI image sequences.

[0004] To further improve the quality of generated images, researchers have recently proposed multimodal fusion methods. For example, Hi-Net combines imaging information from multiple modalities through a multi-level fusion strategy to enhance the quality of the synthesized target image. Furthermore, for breast cancer DCE-MRI image synthesis, CEMRI combines multi-b-value diffusion-weighted imaging (DWI) and T1-weighted imaging, extracting and fusing features through a specific deep learning network, improving image quality in the absence of contrast agents. These studies demonstrate the promising application of deep learning technology in breast cancer MRI image synthesis and conversion tasks.

[0005] While existing technologies have made progress in the field of medical image synthesis, numerous challenges and limitations remain. First, traditional DCE-MRI relies on contrast agents. While these agents can enhance the visualization of lesions, they also carry potential risks such as bioaccumulation and nephrogenic systemic fibrosis. Therefore, reducing or even completely eliminating the use of contrast agents while maintaining high-quality image contrast is a key challenge facing existing technologies.

[0006] Physical modeling-based approaches rely on strict mathematical assumptions, such as tissue homogeneity and first-order kinetic modeling of contrast agent diffusion. However, in practice, the physiological conditions of different patients vary significantly, limiting the applicability of the Tofts model. Furthermore, this approach requires high-temporal-resolution data to accurately fit the dynamic changes of the contrast agent. Acquiring high-temporal-resolution DCE-MRI data in a clinical setting may be limited by the scanning equipment and patient tolerance.

[0007] Current deep learning-based medical image conversion methods still have limitations in breast MRI image synthesis tasks. Supervised learning methods rely on high-quality paired data, but in actual clinical applications, it is difficult to obtain fully matched paired image data. The reliance on strictly registered input-target data pairs results in insufficient generalization of the model in clinical scenarios, and slight displacements caused by patient movement or equipment differences can cause distortion of the synthesized image. Although unsupervised learning methods (such as CycleGAN) can complete image conversion without paired data, due to the lack of strict supervision constraints, the generated images may have artifacts, blurring, or texture loss, affecting the reliability of clinical diagnosis.

[0008] Furthermore, while existing multimodal fusion methods can combine information from different imaging modalities to improve the quality of image synthesis, there is still room for improvement in feature extraction and fusion. For example, some methods fail to adequately address the characteristics of different modalities, resulting in inadequate utilization of the fused information and impacting the authenticity and consistency of the generated image. Therefore, further improving the effectiveness of multimodal fusion strategies to fully utilize information from different imaging modalities and improve the quality of generated images remains a challenge that current technologies need to overcome.

[0009] In general, existing technologies still have certain limitations in reducing contrast agent usage, improving the stability of unsupervised image conversion, and optimizing multimodal feature fusion. To this end, the present invention proposes a method for generating contrast agent-free DCE-MRI images based on multimodal feature fusion. Summary of the Invention

[0010] The technical problem to be solved by the present invention is: how to solve the problems of information loss, incomplete texture details, insufficient utilization of cross-modal information, etc. in the existing technology during the image generation process. A contrast agent-free DCE-MRI image generation method based on multimodal feature fusion is provided. This method achieves high-quality medical image synthesis through the comprehensive application of multi-source feature extraction, perfusion information enhanced attention mechanism and generative adversarial network.

[0011] The present invention solves the above technical problems through the following technical solutions, which include the following steps:

[0012] S1: Data preprocessing

[0013] Preprocess the image data to obtain an input image, which is a plurality of DWI images with different b values and a T1-weighted image;

[0014] S2: Multi-source feature extraction

[0015] The multi-source feature extraction module is used to extract multi-source features from multiple DWI images with different b-values, and the extracted multi-source features are input into the generator of the generative adversarial network.

[0016] S3: Enhanced attention through information infusion

[0017] The pre-trained VGG-19 network is used to extract features from T1-weighted images. The extracted features are then fed into the perfusion information enhanced attention module for processing. The processed results are then fused with the corresponding features of the decoder layer in the generator of the generative adversarial network.

[0018] S4: Image Generation

[0019] DCE-MRI images are generated through the generator of the generative adversarial network, thereby completing the DCE-MRI image generation work without contrast agent.

[0020] Furthermore, in the step S1, the data preprocessing method includes synchronous registration, image normalization, data enhancement and format conversion; wherein, synchronous registration is to synchronously register image data of different scanning methods, image normalization is to normalize all scanned images, the normalization method is the standardization of pixel intensity, data enhancement includes random rotation, random flipping, random scaling and random elastic deformation processing, and format conversion is to store image data in accordance with the DICOM standard format.

[0021] Furthermore, in step S2, the multi-source feature extraction module includes a wavelet transform enhanced feature extraction unit and an apparent diffusion coefficient feature extraction unit; wherein the wavelet transform enhanced feature extraction unit is used to decompose the DWI images with different b values by discrete wavelet transform into a low-frequency sub-band and three high-frequency sub-bands, the low-frequency sub-band retains the overall contour, and the high-frequency sub-band retains the texture and edge information, and then the high-frequency and low-frequency features of the DWI images with different b values are fused, and the final features are obtained by using a convolution layer. The apparent diffusion coefficient feature extraction unit is used to calculate the apparent diffusion coefficients of the DWI images with different b values in pairs, obtain the differential information from low b value to high b value, and then obtain the apparent diffusion coefficient difference map; the feature F obtained by the wavelet transform enhanced feature extraction unit is added to the image. fuse It is combined with the apparent diffusion coefficient difference map obtained by the apparent diffusion coefficient feature extraction unit, concatenated in the channel dimension, and used as the input of the generator.

[0022] Furthermore, in the wavelet transform enhanced feature extraction unit, the specific processing process is as follows:

[0023] S201: using discrete wavelet transform to decompose the DWI images with different b values into low-frequency and high-frequency sub-bands, and then splicing the corresponding sub-bands of the DWI images with different b values along the channel dimension;

[0024] S202: The corresponding sub-bands from the DWI images with different b values are fused together through a 1×1 convolutional layer to obtain sub-band features. Among them, HH, HL, LH and LL represent the high-frequency information of the image in the diagonal direction, the vertical edge information of the image, the horizontal edge information of the image, and the low-frequency information of the image respectively;

[0025] S203: Downsample the input using a 3×3 convolutional layer along the channel dimension to match the subband size obtained by wavelet transform, and subtract the low-frequency subband features obtained by fusion. Further extract high-frequency information to obtain differential high-frequency features F d ;

[0026] S204: Differentiate high frequency features F d and the fused high-frequency sub-band features The frequency fusion unit is sequentially sent to the frequency fusion unit for further feature extraction; in this process, the frequency fusion unit adopts the spatial attention mechanism to focus on the fused high-frequency sub-band features. And the differential high-frequency feature F d The difference between them is weighted to enhance the relevant information of the four high-frequency sub-bands;

[0027] S205: Then perform a 1×1 convolution to obtain the extracted features. Finally, the extracted feature F is obtained by inverse wavelet transform fuse .

[0028] Furthermore, in the apparent diffusion coefficient feature extraction unit, the calculation formula of the apparent diffusion coefficient is as follows:

[0029]

[0030] Where ADC is the apparent diffusion coefficient, S h 、S l Represent high b value b h and low b value b l The signal strength below.

[0031] Furthermore, the perfusion information enhanced attention module includes a spatial attention branch and a channel attention branch, and the specific processing process is as follows:

[0032] S31: In the spatial attention branch, for the input feature f, the convolution layer is first applied to generate components f1, f2 and f3, and components f1 and f2 are converted from R C×H×WReshape into R C×HW ; Then perform matrix multiplication on the transpose of components f2 and f1, and then calculate the attention feature map M∈R through softmax C×C ; A matrix multiplication operation is performed between component f3 and the attention feature map M. The result of the operation is reshaped and added back to the source image f to obtain the output of the spatial attention branch. The source image f is the input feature f;

[0033] S32: In the channel attention branch, for the input feature f, a convolution operation is first performed to generate two different feature representations; then these two feature representations pass through a fully connected layer, activated by the ReLU function, and then pass through another fully connected layer and normalized using the Sigmoid function to obtain attention weights; finally, the attention weights are multiplied by the input feature f channel by channel to obtain an enhanced feature representation, which is the output of the channel attention branch;

[0034] S33: The outputs of the spatial attention branch and the channel attention branch are fused through SumFusion to produce the final attention-enhanced feature representation.

[0035] Furthermore, in the generative adversarial network, the generator adopts an encoder-decoder architecture, and the generator includes four encoder convolution blocks, four decoder convolution blocks, three pooling layers and three upsampling layers. Each convolution block includes a 3×3 convolution layer, a batch normalization layer and a ReLU activation layer, wherein the encoder part gradually extracts features through four encoder convolution blocks and three pooling layers; the decoder performs feature fusion through four decoder convolution blocks, and then gradually restores the spatial resolution of the original image through three upsampling layers; the discriminator includes five discriminator convolution blocks, and the discriminator is a multi-layer serial structure. The layers are connected by discriminator convolution blocks and normalization layers to extract features from the generated image and evaluate the authenticity of the generated image; each discriminator convolution block contains a 3×3 convolution layer, an instance normalization layer and a ReLU activation layer; the output of the generator of the generative adversarial network also needs to be sent to a U-Net-based registration network to obtain the final generation result.

[0036] Furthermore, the generator learns the mapping relationship of images through an encoder-decoder architecture, uses a multi-source feature extraction module to extract DWI image features of different b values, and inputs the features into the encoder part of the generator for further processing; uses the perfusion information enhanced attention module to fuse the features of the T1-weighted image extracted by the VGG-19 network with the features of the corresponding layer in the decoder. After fusion, the decoder reconstructs the image through the upsampling layer.

[0037] Furthermore, when the generative adversarial network is trained, the final generated result is sent to the discriminator, which scores the final generated result of the generator to determine whether the model has converged. The total loss function during training is as follows:

[0038]

[0039] Among them, G represents the generator, D represents the discriminator, R represents the registration network, and L SR is the structure registration loss, L smooth is the smoothing loss, L adv is the generator loss, L D is the discriminator loss, and α, β, γ, and ε are the corresponding weight coefficients.

[0040] Furthermore, in the total loss function, each loss is defined as follows:

[0041] Structural registration loss L SR is defined as follows:

[0042]

[0043] in, is the perturbed target image, T is the random deformation field, represents the resampling operation, X is the input of the generator, y is the target image, and E[·] represents the mathematical expectation;

[0044] Smoothing loss L smooth is defined as follows:

[0045]

[0046] in, Represents the local gradient of the image, that is, the generated image G(X) and the perturbed target image are calculated The difference in gradients between

[0047] Generator loss L adv is defined as follows:

[0048]

[0049] Discriminator loss L D is defined as follows:

[0050]

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] 1. Improve the ability to express multimodal information

[0053] A multi-source feature extraction module is adopted, which includes wavelet transform enhanced feature extraction (WFB) and apparent diffusion coefficient (ADC) feature extraction to enrich the expression ability of different modal information.

[0054] 2. Enhance the ability to identify key areas

[0055] The Perfusion Information Enhanced Attention Module (PIEAM) leverages the complementary information between images of different modalities to enhance the recognition of key areas. This module extracts key feature maps based on the VGG-19 pre-trained model and combines it with an attention mechanism to optimize image details.

[0056] 3. Optimize the detail fidelity of the synthesized image

[0057] During the generation process, the similarity between the generated image and the target image is optimized through the game between the generator and the discriminator of the Generative Adversarial Network (GAN). The generator adopts a multi-level convolutional block structure to enhance the detail fidelity of the generated image, while the discriminator is used to distinguish the generated image from the target image.

[0058] 4. Improve the accuracy of matching results

[0059] A multi-loss optimization strategy is adopted, including adversarial loss, structure registration loss (SSIM), deformation field smoothing loss, etc., to improve the accuracy and quality of the final generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 1 is a flow chart of a method for generating contrast-free DCE-MRI images based on multimodal feature fusion in an embodiment of the present invention;

[0061] Figure 2 Schematic diagram of the structure of the generative adversarial network architecture in an embodiment of the present invention;

[0062] Figure 3 2 is a schematic structural diagram of a multi-source feature extraction module MSFEM according to an embodiment of the present invention;

[0063] Figure 4 2 is a schematic diagram of the structure of the information-infused attention enhancement module in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiment.

[0065] Example 1

[0066] like Figure 1As shown, this embodiment provides a technical solution: a contrast agent-free DCE-MRI image generation method based on multimodal feature fusion, which first uses a multi-source feature extraction module (MSFEM) to decompose the input image into high-frequency and low-frequency forms to extract key information under different modalities. Subsequently, a perfusion information enhanced attention module (PIEAM) is used to enhance key features, and the generated results are optimized through a generative adversarial network (GAN) to make them more realistic and have stronger structural consistency. The method includes the following key steps: multi-source feature extraction, information enhanced attention mechanism, image generation and optimization, etc.

[0067] The specific modules and processing procedures of the contrast-free DCE-MRI image generation method based on multimodal feature fusion are as follows:

[0068] 1. Data preprocessing

[0069] Preprocessing was performed on diffusion-weighted imaging with different b values, unenhanced phase imaging, and enhanced dynamic contrast-enhanced imaging. The preprocessing included registering the imaging results from different scanning methods, normalizing pixel intensity, data enhancement, and format conversion.

[0070] 2. Multi-source Feature Extraction Module (MSFEM)

[0071] Wavelet Transform Refinement Fusion (WFB): Discrete Wavelet Transform (DWT) is a time-frequency analysis method that provides localized information about signals in both the time and frequency domains. By decomposing an image into low-frequency and high-frequency subbands, the image's underlying structure and details can be effectively separated. Low-frequency components preserve the overall contour, while high-frequency components contain rich texture and edge information. By fusing the high- and low-frequency features of multiple DWI images, image information is further refined through 1×1 and 3×3 convolutional layers, avoiding information redundancy.

[0072] Apparent Diffusion Coefficient (ADC) Difference Map: ADC values reflect the microstructural characteristics of tumor tissue and provide critical tissue contrast information. By extracting ADC difference features and combining them with high-frequency features, the tissue contrast of the generated image can be enhanced.

[0073] 3. Perfusion Information Enhanced Attention Module (PIEAM)

[0074] By leveraging the pre-trained VGG-19 network to extract high-level features from T1-weighted images, PIEAM is able to capture the edge, texture, and perfusion properties of images.

[0075] Through the attention mechanism, the generator can focus on key feature areas and avoid unfiltered information noise in skip connections.

[0076] The self-attention component of the feature map is calculated and key features are selectively enhanced through matrix multiplication operations.

[0077] 4. Generative Adversarial Networks (GANs)

[0078] The generator learns the mapping relationship of images through the encoder-decoder architecture. In the encoding stage, features are extracted through multi-level convolution and pooling layers, and in the decoding stage, the image is reconstructed through the deconvolution layer.

[0079] The discriminator uses multi-layer convolution blocks to determine the authenticity of the generated image, helping the generator learn realistic image features.

[0080] PIEAM further enhances cross-modal features in the generator to ensure accurate representation of structure and texture in synthesized images.

[0081] 5. Loss Function and Convergence Strategy

[0082] The adversarial loss includes the generator loss and the discriminator loss to ensure the similarity between the generated image and the target image.

[0083] The structural registration loss corrects the pixel displacement error through the deformation field and improves the image registration accuracy.

[0084] The smoothness loss limits the gradient of the deformation field, preventing unnecessary distortions.

[0085] The combined loss function weighs image content, structure, and geometric consistency to ensure high quality of generated images.

[0086] Through the synergistic effect of multi-source feature extraction, a perfusion-enhanced attention module, and a generative adversarial network, this method effectively extracts and fuses multimodal information, enhancing structural detail and tissue texture in the generated images. Furthermore, an innovative loss function design and convergence strategy ensure the quality and accuracy of the generated images. This method not only achieves higher image fidelity in image synthesis tasks but also provides a stable and interpretable solution for DCE-MRI generation.

[0087] The specific differences and advantages of the present invention compared with the prior art are as follows:

[0088] 1. Improve the ability to express multimodal information

[0089] Traditional methods for processing multimodal medical images typically employ direct feature concatenation or simple full-matching strategies, ignoring the deep, complementary relationships between different modal data. This makes it difficult for the model to fully exploit cross-modal semantic information. For example, some methods perform modality alignment solely through key feature matching or simple feature transformations, making it difficult to capture detailed correspondences between complex images.

[0090] The present invention uses a multi-source feature extraction module (MSFEM) to deeply explore the complementary characteristics of each modality and improve the model's ability to understand multimodal data. MSFEM combines wavelet transform-enhanced feature extraction (WFB) with apparent diffusion coefficient (ADC) feature extraction to extract image structure information at different scales. Wavelet transform can effectively decompose high- and low-frequency features to avoid redundant information interference. At the same time, ADC reflects the microstructure of the tissue and helps to enhance the detail expression of key areas. This multi-level feature extraction method enables the model to accurately capture subtle changes in the lesion area, thereby improving generalization and adaptability.

[0091] 2. The attention mechanism optimizes multimodal image fusion and improves the recognition ability of key areas

[0092] Existing medical image synthesis methods often rely on simple feature concatenation or average pooling when fusing multimodal data. This lack of emphasis on key regional information leads to insufficient contrast and detail in the generated images. In DCE-MRI synthesis tasks, the lack of subtle structural information can weaken lesion features, compromising diagnostic accuracy.

[0093] This paper proposes a perfusion information enhanced attention module (PIEAM), which enhances the key features of the generated image by fusing perfusion information from DWI images with structural information from T1-weighted images. PIEAM uses a pre-trained VGG-19 model to extract image features from different resolution layers and combines channel attention with spatial attention mechanisms to ensure that the model can focus on subtle contrast changes in the image. This mechanism effectively optimizes local details in the generated image, enabling the model to generate high-quality DCE-MRI without contrast agents, improving the clinical usability of the images.

[0094] 3. High-fidelity image synthesis to enhance the authenticity of image structure

[0095] While existing medical image generation methods, such as those based on U-Net or CycleGAN, can generate images, they often suffer from issues such as blurred details and unclear edges. Especially for high-resolution medical images, traditional methods struggle to accurately reconstruct the subtle structure of the lesion, impacting the diagnostic value of the images.

[0096] This paper employs a generative adversarial network (GAN) and optimizes the generator with a multi-level convolutional block structure, resulting in high-fidelity representation of generated images in both global and local detail. The generator utilizes skip connections and a feature pyramid to enhance information representation at different scales, while the discriminator is trained using an adversarial loss to improve the similarity between the generated image and the target image. Furthermore, the present invention incorporates a perfusion information-enhanced attention mechanism to ensure that the model accurately reproduces the dynamic imaging characteristics of DCE-MRI when generating images, thereby enhancing the realism of the images.

[0097] 4. Multiple loss optimization strategies to improve the robustness and stability of the model

[0098] Traditional medical image synthesis methods mostly rely on pixel-level loss functions such as mean square error (MSE) or root mean square error (RMSE), but these loss functions have difficulty evaluating the structural similarity of images, resulting in a lack of contrast and edge sharpness in the generated images at the detail level.

[0099] The present invention adopts a multi-loss optimization strategy, including adversarial loss, structural registration loss, smoothness loss and discriminator loss. This loss combination can enhance the expression of detailed information while ensuring the accuracy of the overall structure of the image, so that the final generated DCE-MRI has higher diagnostic value.

[0100] The details of each type of loss are as follows:

[0101] Adversarial loss: Using GAN training strategies to optimize the statistical distribution between generated and target images, improving the naturalness of images;

[0102] Structural registration loss: A U-Net-based registration network is introduced to optimize the spatial consistency of images and avoid artifacts.

[0103] Smoothness loss: avoids unnatural distortions in the generated image and helps generate smoother and distortion-free images;

[0104] Discriminator loss: ensures the distinction between target images and generated images, further constraining the generator to produce more realistic results.

[0105] 5. Enhance the adaptability and scalability of the model and reduce computing costs

[0106] In existing technologies, variations in scanning equipment or image acquisition parameters often affect the adaptability of the model, resulting in a decrease in the quality of the generated images. For example, differences in imaging characteristics between different MRI devices can limit the model's generalization capabilities, requiring extensive data retraining to adapt to new imaging conditions, which undoubtedly increases computational costs and data requirements.

[0107] The cross-modal adaptive learning strategy proposed in this paper improves the model's adaptability in different imaging environments through multi-scale feature extraction, attention mechanism optimization, and deformation field regularization. Furthermore, the use of lightweight convolutional modules reduces computational complexity, enabling the model to run efficiently with limited computing resources. This optimization strategy not only reduces the cost of model deployment but also enhances its adaptability in various clinical application scenarios, providing broader application potential for contrast-free image synthesis in DCE-MRI.

[0108] Example 2

[0109] This embodiment describes the data preprocessing, multi-source feature extraction module, perfusion information enhanced attention module, and generative adversarial network in Example 1 in more detail as follows:

[0110] 1. Data preprocessing

[0111] Processing of images with different scanning modes: During data processing, image data with different scanning modes are synchronously registered.

[0112] Image normalization: All scan results are normalized, including pixel intensity normalization. Pixel intensity normalization normalizes the image pixel intensity values of different patients through maximum and minimum value normalization to ensure that the pixel distribution of different case data is in the same range. The formula for maximum and minimum value normalization is as follows:

[0113] Data enhancement: including random rotation, random flipping, random scaling, and random elastic deformation.

[0114] Data storage and transmission: After the above processing, the data is uniformly stored in the DICOM standard format to ensure that the data can be transmitted losslessly to other analysis platforms or used for subsequent artificial intelligence training models, facilitating data sharing and further medical research.

[0115] 2. Multi-source feature extraction module

[0116] We use different symbols to represent the diffusion weighted imaging (DWI) with different b values, namely DWI0, DWI 120 , DWI 800 , DWI 1500 The unenhanced phase image and the enhanced phase image (target image) are represented by symbols T1 and DCE2. Wavelet transform is a common and powerful decomposition technique that can divide an image into a low-frequency subband and three high-frequency subbands. The low-frequency subband retains the basic structure of the image, while the high-frequency subband retains the complex details. This transformation uses four sets of filters, namely LL, LH, HL, and HH, to extract these four subbands. The definitions of L and H are as follows:

[0117] Diffusion-weighted imaging observes tissue microstructure information by measuring the diffusion state of water molecules within tissue. Tumor lesions have high cell density and limited extracellular space, which greatly limits the diffusion of water molecules. The apparent diffusion coefficient (ADC) is an indicator used to quantify the diffusion capacity of water molecules within tissue. It can generate differential information based on the diffusion characteristics and enhance the structural and microstructural contrast in the generated image. The formula for the apparent diffusion coefficient (ADC) is as follows: We will DWI0, DWI 120 , DWI 800 , DWI 1500 ADC is calculated pairwise to obtain differential information from low b value to high b value, which is used as part of the input of the generative adversarial network to provide the model with more detailed and discriminative input.

[0118] Since the wavelet transform divides the image into a low-frequency sub-band and three high-frequency sub-bands, and the low-frequency sub-band retains the basic structure and contour of the image, while the high-frequency sub-band retains the complex detail information, we use discrete wavelet transform (DWI) to decompose the diffusion-weighted imaging (DWI) of different b values into low-frequency and high-frequency sub-bands, and then splice the corresponding sub-bands of DWI with different b values along the channel dimension. Then, the corresponding sub-bands of DWI from different b values are fused together through 1×1 convolution to obtain sub-band features. In order to fully and accurately extract the high-frequency details in the input sequence and avoid the redundancy of input information, we downsample the input using 3×3 convolution along the channel dimension to match the subband size obtained by wavelet transform, and subtract the fused low-frequency subband features from it. Further extract high-frequency information to obtain high-frequency features F d Then the high frequency feature F d and the fused high-frequency sub-band features The features are then fed into the frequency fusion unit for further feature extraction. The frequency fusion unit uses the spatial attention mechanism to weight the differences between the fused high-frequency sub-band features and the differential high-frequency features to enhance the information of the four high-frequency sub-bands. The extracted features are then obtained through a 1×1 convolution. Finally, the extracted feature F is obtained by inverse wavelet transform fuse .

[0119] The multi-source feature extraction module extracts the feature F obtained by wavelet transform enhancement feature extraction fuse Combined with the apparent diffusion coefficient difference map, it is concatenated in the channel dimension and used as the input of the generator.

[0120] 3. Infusion Information Enhancement Attention Module

[0121] Unenhanced T1 images share most of the same tissue structure and texture features as enhanced DCE-MRI. In the perfusion-enhanced attention modules, we leverage the complementary features of different modalities using a pretrained VGG-19 network as the backbone. This network takes T1 images as input and extracts features from 64-, 128-, and 256-channel layers. These extracted features, along with the output of a generator convolutional block with matching channel layers, are fed into the corresponding perfusion-enhanced attention modules. The attention mechanism enables these modules to focus on key image features, thereby increasing the model's sensitivity to important information.

[0122] In the perfusion information enhanced attention module, given an R C×H×W dimensional feature f, we first apply a convolutional layer to generate components f1, f2, and f3, and transform f1 and f2 from R C×H×W Reshape into R C×HW Next, perform matrix multiplication on the transpose of f2 and f1, and then calculate the attention feature map M∈R through softmax C×C . A matrix multiplication operation is performed between f3 and M. The result is reshaped and added back to the source image f. In addition to the spatial attention branch, PIEAM also contains a channel attention branch. This branch first performs a convolution operation on the input feature f, and then performs maximum pooling (MaxPool) and average pooling (AvgPool) to generate two different feature representations, namely f1 and f2. Then, these two feature representations pass through a fully connected (FC) layer, activated by the ReLU function, and then pass through another FC layer and normalized using the Sigmoid function to obtain the attention weight f3.

[0123] Finally, the attention weights are multiplied by the input features f by channel to obtain the enhanced feature representation f′. The outputs of the spatial attention branch and the channel attention branch are fused through SumFusion to produce the final attention-enhanced feature representation.

[0124] 4. Generative Adversarial Networks

[0125] The output of the generator also needs to be fed into a U-Net-based registration network to obtain the final generated result. Then, the final generated result is fed into the discriminator, which scores the generator's generated result in the range ∈ [0, 1]. The generator architecture consists of eight convolution blocks (four encoder convolution blocks, four decoder convolution blocks), three pooling layers, and three upsampling layers. Each convolution block includes a 3×3 convolution layer, a batch normalization layer, and a ReLU activation layer. The discriminator consists of five convolution blocks, each of which contains a 3×3 convolution layer, an instance normalization layer, and a ReLU activation layer.

[0126] Loss function and convergence strategy

[0127] a) Loss function design

[0128] We represent the input of the generator as:

[0129] X={DWI0,DWI 120 ,DWI 800 ,DWI 1500 ,T1}

[0130] Generator loss: The adversarial loss aims to ensure that the generated DCE-MRI images are difficult to distinguish from real DCE-MRI images. The generator is optimized against the discriminator to make the generated images more visually realistic. The generator G "fools" the discriminator D by minimizing the following adversarial loss:

[0131]

[0132] Structural registration loss: To ensure that the generated image is anatomically consistent with the target image y(GT, ground truth), we introduce a structural registration loss. By minimizing the displacement error between the generated image and the target image, the generated image can be aligned with the target image in terms of spatial structure. This loss is defined as:

[0133]

[0134] in, is the displacement error, T is the random deformation field, represents the resampling operation, and R is the registration network. This loss ensures the anatomical consistency of the generated image and is optimized by the deformation field T.

[0135] Smoothness loss: To avoid unnatural distortions in the generated image, the smoothness loss ensures that the deformation field T remains smooth and minimizes large gradients in the deformation field. This helps to generate smoother and distortion-free images. The definition of smoothness loss is as follows:

[0136]

[0137] in, Represents the local gradient of the image, that is, the generated image G(X) and the target image The difference in gradients between

[0138] Discriminator loss: The goal of the discriminator D is to distinguish the target image from the generated image. During the training process of the Generative Adversarial Network (GAN), the discriminator improves its discrimination ability by maximizing its loss, and its loss function is defined as:

[0139]

[0140] Total loss: The optimization goal of the generator G and the registration network R is to minimize their total loss, while the goal of the discriminator D is to maximize its loss, thereby achieving adversarial optimization between the generated image and the target image. The total loss function is:

[0141]

[0142] Among them, α, β, γ, and ε are used as weight coefficients.

[0143] b) Convergence strategy

[0144] Gradient Descent: The gradient is calculated through the back-propagation algorithm and the network parameters are updated using an optimization algorithm (such as the Adam optimizer). The generator G and the registration network R optimize their parameters by minimizing the loss function, while the discriminator D optimizes its parameters by maximizing the loss.

[0145] Learning rate scheduling: To avoid oscillations during training, a dynamic learning rate adjustment strategy is used. A higher learning rate is used in the early stages of training to accelerate the training process, and then the learning rate is gradually reduced to refine the model.

[0146] Adversarial training: The generator and discriminator adopt an alternating training strategy, updating the parameters of the generator and discriminator separately in each training step, ensuring that the generator can continuously optimize the quality of generated images while improving the discriminator's ability to distinguish between real and forged images.

[0147] Example 3

[0148] This embodiment further explains the specific network structure and modules of the second embodiment.

[0149] like Figure 2 As shown in Figure 3, our generative adversarial network architecture integrates the apparent diffusion coefficient and wavelet transform via a multi-source feature extraction module (MSFEM) to fuse specific and shared features from multi-b-value DWI images. Before contrast agent injection, T1 MRI features are extracted using VGG-19, and perfusion information is enhanced using an attention module. During training, a registration network minimizes the offset between the generated and target images.

[0150] like Figure 3As shown in Figure 3, our proposed MSFEM consists of two main components: the WFB and the ADC. The proposed wavelet-inspired fusion module (WFB) processes multi-wavelength-valued DWI sequences as input. By applying a wavelet transform, this module enhances the high-frequency features of each sub-band while preserving the shared features of the multi-b-valued DWI sequences. The output effectively enhances both common and b-value-specific features in the fusion results. Due to the high density of cancer cells and the limited extracellular space, water molecule diffusion is significantly restricted, resulting in low ADC values. ADC values provide information about the microstructure of tumor tissue. The ADC generates differential information based on the diffusion characteristics, enhancing structural and microstructural contrast in the generated images. This feature enhancement complements the high-frequency enhancement of the WFB module, providing the model with more detailed and discriminative input.

[0151] like Figure 4 As shown in the figure, the present invention proposes a perfusion information enhanced attention module (PIEAM), which utilizes the blood perfusion information in the unenhanced T1 sequence and the multi-b-value DWI sequence. This ensures that the attention matrix generated by the attention mechanism can effectively focus on the perfusion information of each region, thereby eliminating the dependence on contrast agents.

[0152] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for generating contrast-free DCE-MRI images based on multimodal feature fusion, characterized in that: The following steps are involved: S1: Data preprocessing Preprocess the image data to obtain an input image, which is a plurality of DWI images with different b values and a T1-weighted image; S2: Multi-source feature extraction The multi-source feature extraction module is used to extract multi-source features from multiple DWI images with different b-values, and the extracted multi-source features are input into the generator of the generative adversarial network. S3: Enhanced attention through information infusion The pre-trained VGG-19 network is used to extract features from T1-weighted images. The extracted features are then fed into the perfusion information enhanced attention module for processing. The processed results are then fused with the corresponding features of the decoder layer in the generator of the generative adversarial network. S4: Image Generation DCE-MRI images are generated through the generator of the generative adversarial network, thereby completing the DCE-MRI image generation work without contrast agent.

2. The method for generating contrast-agent-free DCE-MRI images based on multimodal feature fusion according to claim 1, characterized in that: In step S1, the data preprocessing method includes synchronous alignment, image normalization, data enhancement and format conversion; wherein, synchronous alignment is to synchronously align image data of different scanning methods, image normalization is to normalize all scanned images, the normalization method is the standardization of pixel intensity, data enhancement includes random rotation, random flipping, random scaling and random elastic deformation processing, and format conversion is to store image data in accordance with the DICOM standard format.

3. The method for generating contrast-agent-free DCE-MRI images based on multimodal feature fusion according to claim 1, wherein: In step S2, the multi-source feature extraction module includes a wavelet transform enhanced feature extraction unit and an apparent diffusion coefficient feature extraction unit; wherein the wavelet transform enhanced feature extraction unit is used to decompose the DWI images with different b values by discrete wavelet transform into a low-frequency sub-band and three high-frequency sub-bands, the low-frequency sub-band retains the overall contour, and the high-frequency sub-band retains the texture and edge information, and then the high-frequency and low-frequency features of the DWI images with different b values are fused, and the final features are obtained by using a convolution layer. The apparent diffusion coefficient feature extraction unit is used to calculate the apparent diffusion coefficients of the DWI images with different b values in pairs, obtain the differential information from low b value to high b value, and then obtain the apparent diffusion coefficient difference map; the feature F obtained by the wavelet transform enhanced feature extraction unit is added to the image. fuse It is combined with the apparent diffusion coefficient difference map obtained by the apparent diffusion coefficient feature extraction unit, concatenated in the channel dimension, and used as the input of the generator.

4. The method for generating contrast-agent-free DCE-MRI images based on multimodal feature fusion according to claim 3, wherein: In the wavelet transform enhanced feature extraction unit, the specific processing process is as follows: S201: using discrete wavelet transform to decompose the DWI images with different b values into low-frequency and high-frequency sub-bands, and then splicing the corresponding sub-bands of the DWI images with different b values along the channel dimension; S202: The corresponding sub-bands from the DWI images with different b values are fused together through a 1×1 convolutional layer to obtain sub-band features. Among them, HH, HL, LH and LL represent the high-frequency information of the image in the diagonal direction, the vertical edge information of the image, the horizontal edge information of the image, and the low-frequency information of the image respectively; S203: Downsample the input using a 3×3 convolutional layer along the channel dimension to match the subband size obtained by wavelet transform, and subtract the low-frequency subband features obtained by fusion. Further extract high-frequency information to obtain differential high-frequency features F d ; S204: Differentiate high frequency features F d and the fused high-frequency sub-band features The frequency fusion unit is sequentially sent to the frequency fusion unit for further feature extraction; in this process, the frequency fusion unit adopts the spatial attention mechanism to focus on the fused high-frequency sub-band features. And the differential high-frequency feature F d The difference between them is weighted to enhance the relevant information of the four high-frequency sub-bands; S205: Then perform a 1×1 convolution to obtain the extracted features. Finally, the extracted feature F is obtained by inverse wavelet transform fuse .

5. The method for generating contrast-agent-free DCE-MRI images based on multimodal feature fusion according to claim 3, characterized in that: In the apparent diffusion coefficient feature extraction unit, the calculation formula of the apparent diffusion coefficient is as follows: Where ADC is the apparent diffusion coefficient, S h 、S l Represent high b value b h and low b value b l The signal strength below.

6. The method for generating contrast-agent-free DCE-MRI images based on multimodal feature fusion according to claim 3, characterized in that: The perfusion information enhanced attention module includes a spatial attention branch and a channel attention branch. The specific processing process is as follows: S31: In the spatial attention branch, for the input feature f, the convolution layer is first applied to generate components f1, f2 and f3, and components f1 and f2 are converted from R C×H×W Reshape into R C×HW ; Then perform matrix multiplication on the transpose of components f2 and f1, and then calculate the attention feature map M∈R through softmax C×C ; A matrix multiplication operation is performed between component f3 and the attention feature map M. The result of the operation is reshaped and added back to the source image f to obtain the output of the spatial attention branch. The source image f is the input feature f; S32: In the channel attention branch, for the input feature f, a convolution operation is first performed to generate two different feature representations; then these two feature representations pass through a fully connected layer, activated by the ReLU function, and then pass through another fully connected layer and normalized using the Sigmoid function to obtain attention weights; finally, the attention weights are multiplied by the input feature f channel by channel to obtain an enhanced feature representation, which is the output of the channel attention branch; S33: The outputs of the spatial attention branch and the channel attention branch are fused through SumFusion to produce the final attention-enhanced feature representation.

7. The method for generating contrast-agent-free DCE-MRI images based on multimodal feature fusion according to claim 6, characterized in that: In the generative adversarial network, the generator adopts an encoder-decoder architecture. The generator includes four encoder convolution blocks, four decoder convolution blocks, three pooling layers, and three upsampling layers. Each convolution block includes a 3×3 convolution layer, a batch normalization layer, and a ReLU activation layer. The encoder part gradually extracts features through four encoder convolution blocks and three pooling layers. The decoder performs feature fusion through four decoder convolution blocks, and then gradually restores the spatial resolution of the original image through three upsampling layers; the discriminator includes five discriminator convolution blocks. The discriminator has a multi-layer serial structure, and the layers are connected by discriminator convolution blocks and normalization layers to extract features from the generated image and evaluate the authenticity of the generated image; each discriminator convolution block contains a 3×3 convolution layer, an instance normalization layer and a ReLU activation layer; the output of the generator of the generative adversarial network also needs to be fed into a U-Net-based registration network to obtain the final generation result.

8. The method for generating contrast-agent-free DCE-MRI images based on multimodal feature fusion according to claim 7, characterized in that: The generator learns the mapping relationship of images through an encoder-decoder architecture, uses a multi-source feature extraction module to extract DWI image features with different b values, and inputs the features into the encoder part of the generator for further processing; uses the perfusion information enhanced attention module to fuse the features of the T1-weighted image extracted by the VGG-19 network with the features of the corresponding layer in the decoder. After fusion, the decoder reconstructs the image through the upsampling layer.

9. The method for generating contrast-agent-free DCE-MRI images based on multimodal feature fusion according to claim 1, wherein: When the generative adversarial network is trained, the final generated result is sent to the discriminator, which scores the final generated result of the generator to determine whether the model has converged. The total loss function during training is as follows: Among them, G represents the generator, D represents the discriminator, R represents the registration network, and L SR is the structure registration loss, L smooth is the smoothing loss, L adv is the generator loss, L D is the discriminator loss, and α, β, γ, and ε are the corresponding weight coefficients.

10. The method for generating contrast-agent-free DCE-MRI images based on multimodal feature fusion according to claim 9, characterized in that: In the total loss function, each loss is defined as follows: Structural registration loss L SR is defined as follows: in, is the perturbed target image, T is the random deformation field, represents the resampling operation, X is the input of the generator, y is the target image, and E[·] represents the mathematical expectation; Smoothing loss L smooth is defined as follows: in, Represents the local gradient of the image, that is, the generated image G(X) and the perturbed target image are calculated The difference in gradients between Generator loss L adv is defined as follows: Discriminator loss L D is defined as follows:

Images