A method for generating dynamic enhanced image features from T2-weighted image features

Abstract

A method for generating dynamic contrast-enhanced imaging features from T2-weighted imaging features, comprising: acquiring breast cancer T2-weighted imaging; performing breast segmentation on the acquired T2-weighted imaging to obtain breast imaging; extracting T2-weighted imaging features from the breast imaging using a trained convolutional neural network; generating dynamic contrast-enhanced imaging features based on the T2-weighted imaging features using a trained deep generative adversarial network; classifying breast cancer pathological information based on the newly generated dynamic contrast-enhanced imaging features, and the pathological information classification includes: benign and malignant classification, molecular typing classification, etc. Compared with the traditional pathological information classification method directly based on T2-weighted imaging features, the dynamic contrast-enhanced imaging features generated by the present invention have higher sensitivity. Compared with the pathological information classification method based on dynamic contrast-enhanced imaging features, the present invention has low imaging cost and fast scanning time, and is a more inexpensive and rapid diagnostic method.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a method for generating dynamic contrast enhancement image features from T2-weighted image features. Background Art

[0002] Early diagnosis and treatment of breast cancer can effectively reduce the mortality rate of patients and improve their long-term survival rate. Magnetic Resonance Imaging (MRI) examination is one of the most popular diagnostic methods for breast cancer, with multiple imaging parameters, including Dynamic Contrast Enhancement Magnetic Resonance Imaging (DCE-MRI), T2-Weighted Imaging (T2WI), Diffusion-Weighted Imaging (DWI), etc. The imaging with different parameters has its own characteristics. Among them, DCE-MRI obtains high-quality images before and after injecting contrast agents in multiple groups, obtains the morphological and hemodynamic information of lesions, has high sensitivity for the diagnosis of breast cancer, and is clinically commonly used for the staging evaluation and molecular typing evaluation of breast cancer.

[0003] However, the cost of DCE-MRI is high, the scanning time is long, and it requires enhanced injection of contrast agents, which brings great risks to patients intolerant to contrast agents. T2WI is another basic conventional scan in standard MRI examination, which is simple and fast to shoot, does not require injection of contrast agents, and is commonly used to exclude cysts, intramammary lymph nodes and other benign breast lesions, and plays an important role in the clinical diagnosis of breast cancer by doctors. However, compared with DCE-MRI containing richer lesion information, its sensitivity is lower.

[0004] Therefore, researching a method for generating dynamic contrast enhancement image features from T2-weighted image features based on a deep adversarial network, generating new dynamic contrast enhancement (DCE-MRI) image features from T2-weighted (T2WI) image features, using the newly generated dynamic contrast enhancement image features to predict breast cancer pathological information, and improving the pathological diagnosis value of T2-weighted images have important practical value and significance for promoting its application in breast cancer diagnosis and reducing medical costs. Summary of the Invention

[0005] The object of the present invention is to propose a method for generating dynamic contrast enhancement image features from T2-weighted image features based on a deep adversarial network, including the following steps:

[0006] S1: Construct a breast cancer dataset consisting of breast cancer images. Each sample in the breast cancer dataset contains three types of data: DCE-MRI images, T2WI images, and the class labels of the samples.

[0007] S2: Perform data preprocessing on the breast cancer images in the breast cancer dataset. For each DCE-MRI and T2WI image, use breast segmentation technology to segment the breast, remove the thoracic cavity and skin parts in the image, and only retain the unilateral breast with lesions.

[0008] S3: For the breast dataset obtained after the segmentation in S2, splice the images of the six sequences of the DCE-MRI images in the channel dimension to obtain 6-channel DCE-MRI images, and replicate the T2WI images three times in the channel dimension to obtain 3-channel T2WI images.

[0009] S4: Data division. Divide the breast dataset into a training set and a test set by stratified sampling.

[0010] S5: Based on the data in the training set, use the image classification method based on a convolutional neural network to perform breast cancer classification pre-training to obtain a DCE-MRI image feature extractor and a T2WI image feature extractor.

[0011] S6: For the 6-channel DCE-MRI images and 3-channel T2WI images obtained in S3, use the DCE-MRI image feature extractor and the T2 image feature extractor obtained in S5 to extract DCE-MRI image features and T2WI image features respectively.

[0012] S7: For the DCE-MRI image feature dataset and the T2WI image feature dataset obtained in S6, use the feature generation method based on a generative adversarial network to perform DCE-MRI image feature generation training to generate DCE-MRI image features based on T2WI image features.

[0013] S8: On the test set, test the pathological information diagnosis performance of the new DCE-MRI image features generated based on T2WI image features.

[0014] Preferably, the convolutional neural network in S5 is ResNet, where the feature extractor is the backbone network of ResNet, and the classifier is the fully connected layer classifier of ResNet.

[0015] Preferably, the specific steps of the breast cancer classification pre-training are as follows:

[0016] A1: Input the image into the feature extractor and infer to obtain the feature z of the image.

[0017] A2: Input the feature z of the image into the classifier to obtain the predicted value of the image.

[0018] A3: Combine the true label y of the image and calculate the classification loss of the model. Its loss function is:

[0019]

[0020] A4: According to the classification loss L, use the gradient descent method to update the parameters in the feature extractor and classifier.

[0021] A5: Repeat the above steps A1 - A4 and use the early stopping mechanism to retain the parameters during the model training process.

[0022] A6: Due to the sample imbalance in the breast cancer imaging dataset, use AUC as the evaluation index of the model to evaluate the model.

[0023] Preferably, the evaluation method of the AUC is:

[0024]

[0025] where i is a sample belonging to the positive (malignant) class, and rank i means sorting all the samples in the test set by probability from high to low, and sample i ranks at the rank i position, M is the number of positive samples, and N is the number of negative samples.

[0026] Preferably, the step S5 includes the following steps:

[0027] S51: Based on the DCE - MRI image data, use the image classification method based on convolutional neural network to train the DCE - MRI image classification network and obtain the feature extractor and classifier of the DCE - MRI image.

[0028] S52: Based on the T2WI image data, use the image classification method based on convolutional neural network to train the T2WI image classification network and obtain the feature extractor and classifier of the T2WI image.

[0029] Preferably, the generative adversarial network in S7 includes an encoder, a decoder, a generator, and a discriminator. The encoder and decoder form a set of autoencoders, and the generator and discriminator form the generative adversarial network.

[0030] Preferably, the specific training steps of the generative adversarial network are as follows:

[0031] S71: Input the T2WI image features into the autoencoder structure and train the autoencoder. Its loss function is

[0032] L AE = MSE(x, Dec(Enc(x)))

[0033] where MSE is the mean squared error, Dec is the decoder network, Enc is the encoder network, and x is the feature of the T2 image. Based on the loss function L AE , the parameters of the autoencoder network are updated using the Adam optimizer;

[0034] S72: For the T2WI image features, encode them using the encoder to obtain the latent variable z;

[0035] S73: Input the latent variable z into the generator to generate new pseudo DCE-MRI image features X f ;

[0036] S74: Input the real DCE-MRI image features X r and the pseudo DCE-MRI image features X f generated based on the T2WI image features into the discriminator, and calculate the loss function:

[0037] L D = D(X f ) - D(X r )

[0038] where D is the discriminator. Based on the loss function L D , the parameters of the discriminator are updated using the Adam optimizer;

[0039] S75: Fix the parameters of the discriminator D, input the latent variable z into the generator to generate pseudo DCE-MRI image features X f , and calculate the loss function:

[0040] L G = -D(G(z))

[0041] where G is the generator. Based on the loss function L G , the parameters of the generator are updated using the Adam optimizer;

[0042] S76: Repeat the above steps S71 - S75 to update the parameters of the encoder, decoder, generator, and discriminator of the feature generation module until all parameters converge and no longer update, and the training of the generative adversarial network is completed.

[0043] Preferably, the class label of the sample in S1 is the benign and malignant label of breast cancer.

[0044] The present invention uses a convolutional neural network in deep learning to extract image features, designs a generative adversarial network to generate DCE-MRI image features based on T2-weighted image features, and uses the newly generated DCE-MRI image features for pathological information classification.

[0045] Compared with the traditional method of directly classifying pathological information based on a convolutional neural network for T2-weighted images, this method has higher sensitivity; compared with the traditional method of directly classifying pathological information based on a convolutional neural network for DCE-MRI images, this method does not rely on DCE-MRI images, has a low imaging cost, and a fast scanning time, and is a more inexpensive and faster diagnostic method. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments.

[0047] Figure 1 is a structural block diagram of the present invention;

[0048] Figure 2 is a network structure diagram of the image classification method based on a convolutional neural network in the present invention;

[0049] Figure 3 is a network structure diagram of the method for generating DCE-MRI image features based on a generative adversarial network in the present invention;

[0050] Figure 4 is a data flow diagram of the diagnosis of breast cancer pathological information of T2-weighted images based on a deep adversarial network in the test stage of the present invention;

[0051] Figure 5 is an ROC curve of three breast cancer classification methods in the specific implementation case of the present invention, namely the breast cancer classification result of T2-weighted images based on a deep adversarial network (the curve indicated by the legend GAN), the breast cancer classification result of T2WI images based on a convolutional neural network (the curve indicated by the legend T2), and the breast cancer classification result of DCE-MRI images based on a convolutional neural network (the curve indicated by the legend DCE). DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] The following will describe the present invention in detail in combination with the drawings and specific implementation cases of the present invention:

[0053] As Figure 1 shown, the method for generating dynamic enhanced image features based on the T2-weighted image features of a deep adversarial network includes a total of six modules, namely: a data collection module 1, an image data preprocessing module 2, an image classification pre-training module 3, an image feature extraction module 4, a deep feature generation module 5, and an image pathological information prediction module 6.

[0054] The image collection module 1 collects breast cancer image data of patients. Each sample contains three types of data: DCE-MRI images, T2WI images, and the class label of the sample. In this embodiment, the benign and malignant pathological information is selected as the sample label. There are a total of 6 sequences of DCE-MRI images, which are images obtained by scanning at six time periods before and after injecting the contrast agent, and the T2WI image contains one sequence.

[0055] The image preprocessing module 2 performs data preprocessing on the breast cancer image data set collected in module 1, mainly including preprocessing processes such as breast segmentation and channel splicing.

[0056] The image classification pre-training module 3 performs classification training on DCE-MRI images and T2WI images respectively based on a convolutional neural network. Through classification pre-training, an image feature extractor for DCE-MRI images, an image feature classifier for DCE-MRI images, an image feature extractor for T2WI images, and an image feature classifier for T2WI images are provided for the subsequent deep feature generation module.

[0057] The image feature extraction module 4 extracts image features. The feature extractor for DCE-MRI images and the feature extractor for T2WI images provided in module 3 are used to extract the data features of the corresponding images respectively, and an image feature data set is obtained by combining the labels of the samples.

[0058] The deep feature generation module 5 uses GAN to perform DCE-MRI image feature generation training. Using the generation ability of GAN, new DCE-MRI image features are generated based on T2WI image features, and prior knowledge in the DCE-MRI image domain is introduced into the T2WI image features to improve the pathological information diagnosis performance of the T2WI image features.

[0059] The image pathological information prediction module 6 performs pathological information diagnosis tests on the new DCE-MRI image features generated based on T2WI image features, and compares with the classification performance of directly using a convolutional neural network in module 3 to evaluate the effectiveness of the generative adversarial method.

[0060] The sample label selected in this embodiment is the benign and malignant label. Combining with the specific embodiment, the implementation steps of the present invention are as follows:

[0061] Step 1: Collect data to construct a breast cancer benign and malignant dataset. This example contains a total of 246 samples, including 139 benign samples and 107 malignant samples. Each sample contains MRI images of two parameters, DCE-MRI and T2WI. The DCE-MRI images contain 6 sequences, namely the mask sequence S0 before injecting the contrast agent and the enhancement sequences S1 - S5 obtained by scanning every 1 minute after injecting the contrast agent. The T2WI images contain one sequence. For the images of each sequence, only the image at the position of the maximum diameter of the mass is retained. The resolutions of both DCE-MRI images and T2WI images are 448×448;

[0062] Step 2: Perform data preprocessing on the breast cancer image dataset. Use breast segmentation technology to segment the breast, remove the chest cavity and skin in the image, and only retain the unilateral breast with lesions;

[0063] Step 3: For the segmented breast image dataset, for all the cropped breast images, unify the image size to 224×224 by adding black borders around the image, then enlarge the image twice to 448×448. Concatenate the images of the six sequences of DCE-MRI images along the channel dimension to obtain 6-channel DCE-MRI images with a data shape of 448×448×6. Duplicate the T2WI images three times along the channel dimension to obtain 3-channel T2WI images with a data shape of 448×448×3. Set the label value of benign samples to 0 and the label value of malignant samples to 1. At this time, a sample has a total of three types of data: a 6-channel DCE-MRI image, a 3-channel T2WI image, and the benign and malignant label of the sample;

[0064] Step 4: According to the principle of stratified sampling, randomly divide the segmented breast data into a training set and a test set in a ratio of 6:4. After division, the training set contains 147 samples and the test set contains 99 samples;

[0065] Step 5: Based on the data in the training set, according to the Figure 2 image classification method based on convolutional neural network in, perform breast cancer classification pre-training to provide a trained feature extractor and classifier for the subsequent deep feature generation process. The selected convolutional neural network is ResNet, where the feature extractor is the backbone network of ResNet and the classifier is the fully connected layer classifier of ResNet. The specific steps of training are as follows:

[0066] Step A1: Input the image into the feature extractor and infer to obtain the image features z;

[0067] Step A2: Input the image features z into the classifier to obtain the predicted value of the image

[0068] Step A3: Combine with the true label y of the image, and calculate the classification loss of the model using cross-entropy. Its loss function is:

[0069]

[0070] Step A4: According to the classification loss L, use the Adma optimizer to update the parameters in the feature extractor and classifier. Set the initial learning rate of the Adam optimizer to 1e-5, the parameter β1 = 0.9, and the parameter β2 = 0.99;

[0071] Step A5: Repeat the above steps A1 - A4, repeat 3000 times in total. Adjust the learning rate to 0.9 times the original every 50 epochs, and adopt the early stopping mechanism to retain the parameters during the model training process;

[0072] Step A6: Evaluate the model with AUC as the evaluation index of the model. The calculation method of AUC is:

[0073]

[0074] where i is a malignant sample, and rank i means sorting the probabilities of all samples in the test set from high to low, and sample i is ranked at the rank i position, M is the number of malignant samples, and N is the number of benign samples;

[0075] Step 6: Use the Figure 2 classification training method, select ResNet50 as the classification model, and perform classification pre-training based on DCE-MRI image data. After the training is completed, obtain the feature extractor and classifier of DCE-MRI images. Based on ResNet50, the AUC for directly classifying the benign and malignant of DCE-MRI images is 0.903;

[0076] Step 7: Use the Figure 2 classification training method, select ResNet34 as the classification model, and perform classification pre-training based on T2WI image data. After the training is completed, obtain the feature extractor and classifier of T2WI images. Based on ResNet34, the AUC for directly classifying the benign and malignant of T2WI images is 0.797, which is about 0.1 lower than the classification performance of DCE-MRI images.

[0077] Step 8: For all breast image data, use the pre-trained feature extractor to extract the features of DCE-MRI images and T2WI images respectively. The feature dimension of DCE-MRI images is 2048 dimensions, and the dimension of T2WI image features is 512 dimensions.;

[0078] Step 9: Based on the breast imaging feature dataset, use the feature generation method based on the generative adversarial network to perform DCE-MRI imaging feature generation training, and generate DCE-MRI imaging features based on T2WI imaging features.

[0079] As shown in the appendix Figure 3 As shown, the entire deep feature generation network has four modules, namely the encoder, decoder, generator, and discriminator. The encoder and decoder form a set of autoencoders, and the generator and discriminator form a generative adversarial network. The specific training steps of the entire network are as follows:

[0080] Step B1: Input the T2WI imaging features into the autoencoder structure and train the autoencoder. Its loss function is

[0081] L AE = MSE(x, Dec(Enc(x)))

[0082] where MSE is the mean square error, Dec is the decoder network, Enc is the encoder network, and x is the feature of T2. Based on the loss function L AE , use the Adam optimizer to update the parameters of the encoder and decoder. Set the initial learning rate of the Adam optimizer to 1e-5, parameter β1 = 0.9, and parameter β2 = 0.99;

[0083] Step B2: For the T2WI imaging features, use the encoder to encode them to obtain the latent variable z;

[0084] Step B3: Input the latent variable z into the generator to generate new pseudo DCE-MRI imaging features X f ;

[0085] Step B4: Input the real DCE-MRI imaging features X r and the pseudo DCE-MRI imaging features X f generated based on the T2WI imaging features into the discriminator and calculate the loss function:

[0086] L D = D(X f ) - D(X r )

[0087] where D is the discriminator. Based on the loss function L D , use the Adam optimizer to update the parameters of the discriminator. Set the initial learning rate of the Adam optimizer to 1e-5, parameter β1 = 0.9, and parameter β2 = 0.99;

[0088] Step B5: Fix the parameters of the discriminator D, input the latent variable z into the generator to generate pseudo DCE-MRI image features X f , calculate the loss function:

[0089] L G = -D(G(z))

[0090] where G is the generator. Based on the loss function L G , use the Adam optimizer to update the parameters of the generator and the encoder. Set the initial learning rate of the Adam optimizer to 1e-5, parameter β1 = 0.9, and parameter β2 = 0.99;

[0091] Step B6: Repeat the above steps B1 to B5 to update the parameters of the encoder, decoder, generator, and discriminator of the feature generation module until all parameters converge and are no longer updated. The feature generation method is thus trained;

[0092] Step 10: According to the structure in Appendix Figure 4 , test the effectiveness of the deep feature generation method based on the test set. For the T2WI image features of a sample in the test set, use the encoder to extract its latent variable z, and input the latent variable into the generator to generate pseudo DCE-MRI image features X f , use the pre-trained DCE-MRI image feature classifier in Step 6 to classify X f , and output the probability of the benign and malignant nature of X f . After the feature generation training, the AUC of the breast cancer benign and malignant classification of the T2WI image features is 0.846, which is 0.049 higher than the original 0.797 in terms of AUC, and the model performance is improved by 6.15%.

[0093] Appendix Figure 5The ROC curves and corresponding AUC values of three breast cancer classification methods in the specific implementation cases of the present invention are the breast cancer classification results of generating DCE-MRI image features based on the T2WI image features of GAN (the curve indicated by GAN), the breast cancer classification results of T2WI images based on a convolutional neural network (the curve indicated by T2), and the breast cancer classification results of DCE-MRI images based on a convolutional neural network (the curve indicated by DCE). It can be seen that when using the DCE-MRI image features generated based on GAN for breast cancer classification, its ROC curve is closer to the ROC curve of DCE-MRI images compared with the model directly using a convolutional neural network for classification. The P-value obtained by performing a significance test on the AUC of the GAN model and the convolutional classification model using the Bootstrap method is 0.0239, that is, the AUC of the GAN model is significantly greater than the AUC of the convolutional model. The classification effect of using the newly generated DCE-MRI image features for benign and malignant classification is significantly improved compared with directly using the T2WI image features for classification.

[0094] The present invention adopts a method for generating dynamic enhanced image features based on the T2-weighted image features of a deep adversarial network, which greatly improves the diagnostic performance of breast cancer case information in T2WI images, further enhances the value of T2WI images in the clinical diagnosis of breast cancer, and thus forms a more low-cost and faster breast cancer diagnosis mode.

[0095] The above specific example ways are used to explain the present invention, rather than limiting the present invention. Any modifications and changes made to the present invention within the spirit and protection scope of the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A method for generating dynamic contrast-enhanced imaging features from T2-weighted imaging features, characterized in that, It includes the following steps: S1: Construct a breast cancer dataset composed of breast cancer images. Each sample in the breast cancer dataset contains three types of data: DCE-MRI images, T2WI images, and the class labels of the samples; S2: Perform data preprocessing on the breast cancer images in the breast cancer dataset. For each DCE-MRI and T2WI image, use breast segmentation technology to segment the breast, remove the thoracic cavity and skin parts in the image, and only retain the unilateral breast with lesions; S3: For the breast dataset obtained after the segmentation in S2, splice the images of the six sequences of the DCE-MRI images in the channel dimension to obtain 6-channel DCE-MRI images, and duplicate the T2WI images three times in the channel dimension to obtain 3-channel T2WI images; S4: Data division. Divide the breast dataset into a training set and a test set through stratified sampling; S5: Based on the data in the training set, use an image classification method based on a convolutional neural network to perform breast cancer classification pre-training to obtain a DCE-MRI image feature extractor and a T2WI image feature extractor; S6: For the 6-channel DCE-MRI images and 3-channel T2WI images obtained in S3, use the DCE-MRI image feature extractor and T2WI image feature extractor obtained in S5 to extract DCE-MRI image features and T2WI image features respectively; S7: For the DCE-MRI image feature dataset and T2WI image feature dataset obtained in S6, use a feature generation method based on a deep generative adversarial network to perform DCE-MRI image feature generation training to generate DCE-MRI image features based on T2WI image features; S8: On the test set, test the pathological information diagnosis performance of the new DCE-MRI image features generated based on T2WI image features; The specific steps of the breast cancer classification pre-training are as follows: A1: Input the image into the feature extractor and infer the feature z of the image; A2: Input the feature z of the image into the classifier to obtain the predicted value of the image A3: Combine the true label y of the image and calculate the classification loss of the model. Its loss function is: A4: According to the classification loss L, use the gradient descent method to update the parameters in the feature extractor and the classifier; A5: Repeat the above steps A1-A4 and use an early stopping mechanism to retain the parameters in the model training process; A6: Due to the sample imbalance in the breast cancer image dataset, use AUC as the evaluation index of the model to evaluate the model. The evaluation method of AUC is: where i is a positive sample, and rank i means sorting all the samples in the test set by probability from high to low, and sample i is ranked at the rank i position, M is the number of positive samples, and N is the number of negative samples; The step S5 includes the following steps: S51: Based on the DCE-MRI image data, use an image classification method based on a convolutional neural network to train a DCE-MRI image classification network to obtain a DCE-MRI image feature extractor and a classifier; S52: Based on the T2WI image data, use an image classification method based on a convolutional neural network to train a T2WI image classification network to obtain a T2WI image feature extractor and a classifier; The generative adversarial network in S7 includes an encoder, a decoder, a generator, and a discriminator. The encoder and the decoder form a set of autoencoders, and the generator and the discriminator form a generative adversarial network; The specific training steps of the generative adversarial network are as follows: S71: Input the T2WI image features into the autoencoder structure and train the autoencoder, with its loss function being L AE = MSE(x, Dec(Enc(x))) where MSE is the mean square error, Dec is the decoder network, Enc is the encoder network, x is the feature of the T2WI image, and based on the loss function L AE , the parameters of the autoencoder network are updated using the Adam optimizer; S72: For the T2WI image features, use the encoder to encode them to obtain the latent variable z; S73: Input the latent variable z into the generator to generate new pseudo DCE-MRI image features X f ; S74: Input the real DCE-MRI image feature X r and the pseudo DCE-MRI image feature X generated based on the T2WI image feature f into the discriminator and calculate the loss function: L D = D(X f ) - D(X r ) Among them, D is the discriminator, based on the loss function L D , and the Adam optimizer is used to update the parameters of the discriminator; S75: Fix the parameters of the discriminator D, input the latent variable z into the generator to generate the pseudo DCE-MRI image feature X f , and calculate the loss function: L G = -D(G(z)) Among them, G is the generator, based on the loss function L G , and the Adam optimizer is used to update the parameters of the generator; S76: Repeat the above steps S71 - S75 to update the parameters of the encoder, decoder, generator, and discriminator of the feature generation module until all parameters converge and are no longer updated, and the training of the generative adversarial network is completed.

2. The method for generating dynamic enhanced image features from T2-weighted image features according to claim 1, wherein, The convolutional neural network in S5 is ResNet, where the feature extractor is the backbone network of ResNet, and the classifier is the fully connected layer classifier of ResNet.

3. A method for generating dynamic enhanced image features from T2-weighted image features according to any one of claims 1 to 2, characterized in that, The class label of the sample in S1 is the benign and malignant label of breast cancer.

Images