A method for dynamically enhancing magnetic resonance image features to generate pathological image features

Abstract

The present invention discloses a method for dynamically enhancing magnetic resonance imaging features to generate pathological image features. The method first obtains DCE-MRI data of breast cancer and corresponding pathological image data, constructs a breast cancer data set, and performs data preprocessing. Secondly, a feature extraction network based on a visual self-attention model and a cross-attention mechanism is constructed to extract DCE-MRI features and pathological image features. Then, based on a generative adversarial network, pathological image features are generated using DCE-MRI features. Finally, using DCE-MRI data, pathological image features are generated through the DCE-MRI branch of a trained feature extraction network and a trained feature generation network. The present invention better utilizes the global information of the image, and enables the model to generate pathological image features through DCE-MRI images even in the absence of pathological images, thereby solving the problem of missing pathological images.

Description

Technical Field

[0001] The present invention belongs to the field of medical imaging and intelligent medical care, and specifically relates to a method for dynamically enhancing magnetic resonance imaging features to generate pathological image features. Background Art

[0002] Artificial intelligence methods can extract and generate features from different images, providing doctors with auxiliary information that is crucial for patient treatment. Magnetic resonance imaging (MRI), which is non-invasive and does not involve ionizing radiation, has high sensitivity in detecting breast lesions and can assist breast surgeons in planning preoperative local staging for breast-conserving surgery and assessing treatment prognosis.

[0003] Imaging is currently the preferred method for breast cancer screening. MRI offers the advantages of high sensitivity, no nuclear radiation during the examination, and high tissue contrast. Dynamic contrast enhancement magnetic resonance imaging (DCE-MRI) has high temporal resolution and is particularly effective for examining dense breasts, making it suitable for Chinese women. Existing medical image feature extraction models and methods mostly require manual feature definition. Deep learning-based feature extraction models are primarily focused on extracting features from single-modality images, and the extracted features fail to incorporate information from other modalities. Multimodal image features contain richer disease information, but require patients to undergo multiple imaging examinations, which is time-consuming and costly, and some examinations, such as pathology, are invasive. Due to methodological limitations, few studies have focused on leveraging information from different images to guide feature extraction. Furthermore, when images from one modality are missing, it is difficult to extract information from other modalities. Summary of the Invention

[0004] The present invention proposes a method for dynamically enhancing magnetic resonance imaging features to generate pathological image features. Based on DCE-MRI and pathological images of breast cancer, the method uses cross-attention to guide information between different modalities to mutually guide feature extraction. A feature generation module uses DCE-MRI features to generate pathological image features. By cross-training real pathological features with simulated pathological features, the simulated pathological image features generated by DCE-MRI features can replace real pathological image features, providing additional auxiliary information for the diagnosis and treatment of breast cancer. The model is guided by pathological image information to extract DCE-MRI image features, and then the DCE-MRI features with pathological image information are used to generate pathological image features. After the model is trained, DCE-MRI features are extracted from DCE-MRI data and simulated pathological image features are generated.

[0005] A method for dynamically enhancing magnetic resonance imaging features to generate pathological image features comprises the following steps:

[0006] Step 1: Obtain breast cancer DCE-MRI data and corresponding pathological image data to construct a breast cancer dataset;

[0007] The breast cancer dataset is collected from hospital clinical data. All patient DCE-MRI data were acquired using the same MRI scanner with a field strength of 3.0 Tesla. Pathology images were collected from the pathology reports of breast cancer patients before neoadjuvant chemotherapy, ensuring that the DCE-MRI and pathology images were obtained at similar times. Information on the efficacy of neoadjuvant chemotherapy was obtained from the patient's pathology report and hospitalization history.

[0008] Step 2: Preprocessing of DCE-MRI data and pathological image data;

[0009] For DCE-MRI data, all patient DCE-MRI voxels were resampled to the same size. For pathology images, a pre-trained segmentation network (Context Pyramid Fusion Network, CPFNet) was used to remove the chest cavity and outer breast skin, retaining only the breast from the pathological biopsy. Pathology image staining was also normalized.

[0010] Step 3: Build a feature extraction network based on the Vision Transformer (ViT) model and the Cross-Attention (CA) mechanism to extract dynamic enhanced MRI image features and pathological image features, and train the feature extraction network;

[0011] The feature extraction network based on the visual self-attention model and the cross-attention mechanism includes a DCE-MRI branch and a pathological image branch:

[0012] The DCE-MRI branch takes DCE-MRI data as input. It first divides the DCE-MRI into DCE-MRI blocks of the same size, then inputs the DCE-MRI blocks into the 3D convolutional layer and adds position encoding to obtain the DCE-MRI block label Patch. MR , followed by adding the randomly initialized DCE-MRI classification label CLS MR Together they serve as the input to the Transformer encoder in the DCE-MRI branch.

[0013] The pathological image branch takes the pathological image as input, first divides the pathological image into pathological image blocks of the same size, and then obtains the pathological image block label Patch through 2D convolution layer and position encoding. path , followed by randomly initialized pathological image classification labels CLS pat Together they serve as the input to the Transformer encoder of the pathological image branch.

[0014] Furthermore, a cross attention module is added between the corresponding Transformer encoders of the DCE-MRI branch and the pathological image branch to guide the DCE-MRI branch and the pathological image branch to learn the information of the other branch. Specifically, the cross attention module uses the CLS of the pathological branch in the DCE-MRI branch. path Patch with DCE-MRI branch MR Combine multiple self-attention operations. Similarly, use CLS of the DCE-MRI branch in the pathology branch. MR Patch with pathology branch path Combined multi-head self-attention operation. Several Transformer encoders and cross-attention modules in the DCE-MRI branch and the pathology branch are stacked to extract deep features of various images. The final CLS MR As a DCE-MRI feature, CLS path As pathological image features.

[0015] Step 4: Based on the generative adversarial network, DCE-MRI features are used to generate pathological image features and the generative adversarial network is trained.

[0016] First, the DCE-MRI features extracted from the DCE-MRI branch in step three are first encoded into latent variables through an encoder, and then the latent variables are used as input to the generator to generate simulated pathological image features. The pathological image features extracted from the pathological image branch in step three and the simulated pathological image features are then used to train the discriminator together. The training is completed when the discriminator cannot correctly classify the simulated pathological image features and the pathological image features extracted from the pathological image branch.

[0017] Step 5: Using DCE-MRI data, pathological image features are generated through the DCE-MRI branch of the trained feature extraction network and the trained feature generation network.

[0018] First, the DCE-MRI data is input into the feature extraction network trained in step 3, and only the DCE-MRI branch is used to extract the DCE-MRI features. The extracted DCE-MRI features are then input into the encoder trained in step 4 to obtain latent variables, and then the generator is used to generate pathological image features.

[0019] Furthermore, a fully connected layer can be added after the feature extraction and generation networks as a classification head to classify breast cancer chemotherapy information. The model can be trained end-to-end. During training, each module is trained simultaneously. To ensure that the model can better generate pathological features to predict disease information, simulated pathological features can be cross-trained with branch pathological features of pathological images. The optimizer and scheduler are used to optimize model parameters and adjust the learning rate.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] Using this technical solution, a Transformer-based model can better utilize global image information. Furthermore, the Transformer does not rely on convolutional layers during feature extraction, making it easier to align features between 3D DCE-MRI and 2D pathology images than traditional CNN models. Furthermore, compared to directly concatenating features, using cross-attention to fuse DCE-MRI and pathology image features allows for interaction between the two modalities during feature extraction, thereby extracting deeper fused features. This allows the two modalities to learn from each other, allowing DCE-MRI features to generate pathology features.

[0022] To address the problem that pathological images are difficult to obtain and invasive, a module was designed to generate pathological image features based on DCE-MRI features. By setting hyperparameters, the model was cross-trained using the generated simulated pathological features and real pathological features, so that the model can generate pathological image features even in the absence of pathological images, thus making up for the problem of missing pathological images.

[0023] Compared to existing medical image feature extraction methods, this solution can extract features that include both global tumor information from DCE-MRI and local pathology information from pathology images. Through a feature generation module and cross-training, DCE-MRI features are used to generate pathology image features, addressing the issue of missing pathology images. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 It is a structural block diagram of the method of the present invention;

[0026] Figure 2 This is a diagram of the feature extraction network structure based on Transformer and cross attention in the present invention;

[0027] Figure 3 Flowchart of the cross-attention module in the DCE-MRI branch in step 3 of the present invention;

[0028] Figure 4 Generate a network structure diagram for the adversarial process in step 4 of the present invention;

[0029] Figure 5 This is a data flow diagram for generating pathological image features using DCE-MRI in step five of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] The structure of a method for dynamically enhancing magnetic resonance image features to generate pathological image features is as follows: Figure 1 As shown, the specific implementation method is described as follows.

[0032] The specific contents of step 1 are as follows:

[0033] Breast cancer clinical data were collected from the hospital. All patients' DCE-MRI data were acquired from the same 3.0T magnetic resonance imaging (MRI) scanner. Pathology images were collected from the breast cancer patients' pathology reports, ensuring that the DCE-MRI and pathology images were obtained at similar times. Information such as the patients' neoadjuvant chemotherapy efficacy was obtained from their pathology reports and hospitalization records as labels. The dataset was divided into training and validation sets in a 7:3 ratio.

[0034] The specific contents of step 2 are as follows:

[0035] The DCE-MRI data were preprocessed using the SimpleITK toolkit. First, the voxel size of the DCE-MRI images for all patients was resampled to 0.65 mm × 0.65 mm × 2 mm. A pretrained Unified Network was used to segment the breast, removing the thoracic cavity and outer breast skin, leaving only the breast undergoing pathological biopsy. This segmentation was saved as a 256 x 256 x 48 . A three-channel 3D image consisting of the start, middle, and end sequences of the DCE-MRI was used as input for the DCE-MRI branch in step three. The pathology images were normalized using the Vahadane algorithm and saved as PNG files, which served as input for the pathology image branch in step three.

[0036] During the training phase, data augmentation is performed on the images. For DCE-MRI data, this includes horizontal and vertical mirroring, rotation, and adding a random bias field. For pathology images, this includes horizontal and vertical mirroring, rotation, and random cropping.

[0037] The specific contents of step three are as follows:

[0038] The feature extraction network based on the ViT model includes a DCE-MRI branch and a pathological image branch. Figure 2 As shown in Figure 2, for the DCE-MRI branch, the input DCE-MRI data is first divided into DCE-MRI blocks of size 16×16×16, which are input into 3D convolution to obtain the DCE-MRI block embedding (PatchEmbedding), and the corresponding position code is added to the PatchEmbedding to obtain the DCE-MRI block tag (PatchToken), which is recorded as Patch MR , each Patch MR The vector length is 96, and a random initialization and Patch MR The classification token with the same vector length as CLS MR ; CLS MR and Patch MR Combination F MR , jointly describe DCE-MRI, where CLS MR Represents the global information of DCE-MRI, Patch MR Represents the local information of DCE-MRI.

[0039] For the pathological image branch, the input pathological image data is divided into 16×16 image blocks, and the Patch Embedding of the pathological image is obtained through the 2D convolution layer and the position encoding is added to obtain the pathological image Patch Token, which is recorded as Patch path , each Patchpath The vector length is 192, and a random initialization and Patch path A vector of the same length as the classification label CLS path ; CLS path and Patch path Combination F path , jointly describe the pathological image, where CLS path Represents the global information of pathological images, Patch path Represents the local information of DCE-MRI.

[0040] F MR As the input of the DCE-MRI branch Transformer encoder, F path Serves as input to the Transformer encoder for the pathology image branch.

[0041] Furthermore, a cross attention module is added between the corresponding Transformer blocks of the two branches to guide the DCE-MRI branch and the pathological image branch to learn the information of the other branch. Figure 3 As shown, first, the projection layer composed of the fully connected layer converts the CLS of DCE-MRI MR Projection onto pathological image CLS path The dimension of Will Patch with pathology branch path Combination, denoted as Later in and Do cross-attention operation (CA) between them. That is,

[0042]

[0043]

[0044]

[0045] in is a trainable parameter, C is the length of the Token vector, and h is the number of attention heads, which is set to 6. Finally, CLS′ MR With Patch MR The combination is used as the input of the next Transformer encoder in the DCE-MRI branch. Similarly, the CLS of the pathology branch is used in the pathology branch. path Patch with DCE-MRI branch MR Perform cross-attention operations.

[0046] Each branch of the feature extraction network contains a total of 9 Transformer encoders, and a cross-attention module is added between every 3 Transformer blocks. Finally, the DCE-MRI branch obtains F′ MR CLS in MR As the DCE-MRI feature, the pathological image branch obtains F′ path CLS in path As pathological image features.

[0047] The specific contents of step 4 are as follows:

[0048] As attached Figure 4 As shown in the figure, the pathological image feature adversarial generative network consists of three modules: encoder, generator, and discriminator. The specific steps of the entire network are as follows:

[0049] 1. Randomly initialize the parameters of the generator G and the discriminator D;

[0050] 2. For the DCE-MRI features output by the DCE-MRI branch in step 3, input them into the encoder for encoding to obtain the latent variable z;

[0051] 3. Input the hidden variable z into the generator to generate the simulated pathological image feature X f ;

[0052] 4. Extract the real pathological image features X r (Pathological image features extracted by the pathological image branch) and simulated pathological image features generated based on DCE-MRI features are input into the discriminator to calculate the loss function:

[0053] L D =D(X f )-D(X r )

[0054] Where D is the discriminator, according to the loss function L D , the parameters of the discriminator are updated using the Adamw optimizer, with an initial learning rate of 5e-4, parameter β1 = 0.9, and parameter β2 = 0.999;

[0055] 5. Fix the parameters of the discriminator D and input the hidden variable z into the generator to generate the simulated pathological image feature X f , calculate the loss function:

[0056] L G =-D(G(z))

[0057] Where G is the generator, according to the loss function L G, the parameters of the discriminator are updated using the Adamw optimizer, with an initial learning rate of 5e-4, parameter β1 = 0.9, and parameter β2 = 0.999;

[0058] Repeat the above steps for training and update the parameters of the encoder, generator, and discriminator of the pathological image feature generation module until convergence.

[0059] The specific contents of step five are as follows:

[0060] According to the attached Figure 5 The structure generates pathological image features based on DCE-MRI. For a sample of DCE-MRI, the DCE-MRI image features are first extracted through the DCE-MRI branch of the feature extraction network in step 3, and then the DCE-MRI features are input into the encoder to obtain the latent variable z. Finally, the latent variable is input into the generator to obtain the simulated pathological image feature X. f .

[0061] Furthermore, the pathologic complete response (pCR) in neoadjuvant chemotherapy information was classified based on the generated simulated pathology image features to verify the effectiveness of the pathology image feature generation method. The ViT network's classification head classifier used DCE-MRI features and pathology image features generated from DCE-MRI data as input. The classifier was trained using the training set for 100 batches, and the discriminator parameters were updated using the Adamw optimizer with an initial learning rate of 5e-4, parameters β_1 = 0.9, and parameters β_2 = 0.999. For comparison, a classifier was also trained using DCE-MRI features and real pathology image features.

[0062] Taking AUC as the evaluation indicator of the model, the calculation method of AUC is:

[0063]

[0064] Where M is the number of negative samples, N is the number of positive samples, i represents one of the negative samples, rank i Indicates that the probability of all samples in the test set is sorted from high to low.

[0065] Finally, on the validation set, the AUC of the classifier trained using DCE-MRI features and pathological image features generated by DCE-MRI data was 0.829, which was 0.134 and 0.085 higher than the classifier trained using only DCE-MRI features (AUC of 0.695) and the classifier trained using only pathological image features (AUC of 0.744), respectively.

[0066] The present invention proposes a method for generating pathological image features from DCE-MRI features. It should be noted that what is disclosed herein is only a representative embodiment. The present invention is not limited to the specific method described herein, and may also have other implementation methods or combinations of other implementation methods.

Claims

1. A method for dynamically enhancing magnetic resonance imaging features to generate pathological image features, characterized in that: The steps include: Step 1: Obtain dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) data of breast cancer and corresponding pathological image data to construct a breast cancer dataset; Step 2: Preprocessing of DCE-MRI data and pathological image data; Step 3: Build a feature extraction network based on the visual self-attention model and cross-attention mechanism to extract dynamic enhanced magnetic resonance imaging features and pathological image features, and train the feature extraction network; The visual self-attention model includes a DCE-MRI branch and a pathological image branch; The DCE-MRI branch takes DCE-MRI data as input, first divides the DCE-MRI into DCE-MRI blocks of the same size; secondly, the DCE-MRI blocks are input into the 3D convolutional layer and position coding is added to obtain the DCE-MRI block label Path MR , followed by adding the randomly initialized DCE-MRI classification label CLS MR Together as the input of the Transformer encoder in the DCE-MRI branch; The pathological image branch takes the pathological image as input, first divides the pathological image into pathological image blocks of the same size; secondly, obtains the pathological image block label Patch through 2D convolution layer and position encoding. path , followed by randomly initialized pathological image classification labels CLS path Together they serve as the input to the Transformer encoder of the pathological image branch; A cross-attention module is added between the corresponding Transformer encoders of the DCE-MRI branch and the pathology image branch; the cross-attention module is specifically as follows: in the DCE-MRI branch, the CLS of the pathology branch is used path Patch with DCE-MRI branch MR Combine multiple self-attention operations. Similarly, use GLS of the DCE-MRI branch in the pathology branch. MR Patch with pathology branch path Combine multiple self-attention operations; The DCE-MRI branch and the pathology branch are composed of several Transformer encoders and cross-attention modules stacked together to extract the deep features of various images; the final GLS MR As a DCE-MRI feature, CLS path as pathological image features; Step 4: Based on the generative adversarial network, DCE-MRI features are used to generate pathological image features and the generative adversarial network is trained. Step 5: Using DCE-MRI data, pathological image features are generated through the trained feature extraction network and adversarial generative network.

2. The method for generating pathological image features by dynamically enhancing magnetic resonance imaging features according to claim 1, characterized in that: The step 1 includes: all patients' DCE-MRI data are collected from the same magnetic resonance imaging acquisition device, and pathological images are collected from the pathology reports of breast cancer patients before neoadjuvant chemotherapy, and the patients' neoadjuvant chemotherapy efficacy information is obtained through the patients' pathology reports and hospitalization reports.

3. The method for generating pathological image features by dynamically enhancing magnetic resonance imaging features according to claim 2, characterized in that: The specific operations of data preprocessing in step 2 are as follows: 2.

1. For DCE-MRI data, resample all patients’ DCE-MRI voxels to the same size; 2.

2. For pathological images, the pre-trained segmentation network CPFNet is used to remove the chest cavity and the skin outside the breast, retaining only the breast on the pathological biopsy side, and the pathological images are stained and normalized.

4. The method for generating pathological image features by dynamically enhancing magnetic resonance imaging features according to claim 3, characterized in that: The specific process of step four is: 4.

1. Use the DCE-MRI features extracted from the DCE-MRI branch in step 3 and encode the DCE-MRI features into latent variables through the encoder; 4.

2. Use the latent variable as the input of the generator to generate simulated pathological image features. Then use the pathological image features extracted by the pathological image branch in step 3 and the simulated pathological image features to train the discriminator. When the discriminator cannot correctly classify the simulated pathological image features and the pathological image features extracted by the pathological image branch, the training is completed.

5. The method for generating pathological image features by dynamically enhancing magnetic resonance imaging features according to claim 4, characterized in that: The specific process of step five is as follows: 5.

1. Input the DCE-MRI data into the feature extraction network trained in step 3 and use the DCE-MRI branch to extract DCE-MRI features. 5.

2. The extracted DCE-MRI features are input into the encoder trained in step 4 to obtain latent variables, and then the generator is used to generate pathological image features.

6. The method for generating pathological image features by dynamically enhancing magnetic resonance imaging features according to claim 5, characterized in that: It also includes adding a fully connected layer as a classification head after the feature extraction network and the generative network to classify breast cancer chemotherapy information; During the training process, a cross-training method is adopted for the simulated pathological features and the pathological features extracted from the pathological image branches.

Images