A cross-modality breast tomographic image feature generation method

Abstract

The application discloses a kind of cross-modal breast tomographic image feature generation methods, comprising: collecting breast cancer molybdenum target image, carries out lesion region segmentation;Lesion region is extracted using the trained convolutional neural network;Based on molybdenum target image feature, generate breast tomographic image feature;The benign and malignant classification of breast cancer is carried out to the generated breast tomographic image.Compared with the traditional benign and malignant classification method directly using molybdenum target image, the breast tomographic image feature generated by the application has better AUC value, relative to the benign and malignant classification based on breast tomographic image, the input image of the application is more widely used, and it is a more accurate and universal diagnostic method.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a cross-modal breast tomographic image feature generation method. BACKGROUND

[0002] Breast cancer seriously threatens human life and health. Early screening of breast cancer can effectively reduce the mortality rate of breast cancer. Mammography (MG) has the characteristics of convenient shooting, non-invasiveness and low cost, and is one of the most commonly used images for early diagnosis of breast cancer in China at present. However, the MG image has poor judgment ability for lesions in dense breast tissue. Digital breast tomosynthesis (DBT) provides structural information of the breast from different image layers, and provides more lesion information. Studies have shown that doctors can obtain a lower misdiagnosis rate when using DBT images in diagnosis compared with MG images. However, due to the limitations of DBT image shooting conditions in China, DBT images cannot be widely promoted in early screening.

[0003] Therefore, the application aims to improve the early diagnosis value of MG images for breast cancer, and designs a generative adversarial network model to generate DBT image features with higher information content, so as to improve the classification value of MG images by means of DBT images with high information content, which has important practical value and significance for promoting the application of the application in breast cancer screening and reducing medical costs. SUMMARY

[0004] The application aims to improve the early diagnosis value of MG images for breast cancer, and designs a generative adversarial network model to generate DBT image features with higher information content, so as to improve the classification value of MG images by means of DBT images with high information content, which has important practical value and significance for promoting the application of the application in breast cancer screening and reducing medical costs.

[0005] The application provides a cross-modal breast tomographic image feature generation method, which comprises the following steps:

[0006] Step 1: Construct a breast cancer data set, which contains DBT images, MG images and benign and malignant labels of breast cancer.

[0007] Step 2: Data preprocessing is performed on the breast cancer images. For the breast cancer images, the lesion area is extracted by using the doctor's annotation information, and conventional data augmentation is performed.

[0008] Step A1: A full zero matrix with the same shape and size as the original image is generated

[0009] Step A2: According to the annotation information, the full zero matrix in the lesion area is set to 1

[0010] Step A3: Use a rectangle to find the boundary and locate the longest side along the x-axis or y-axis as the side length of the square.

[0011] Step A4: Adjust the entire rectangle along its shorter side to ensure that the center of the square mask remains the same as the center of the original rectangular mask.

[0012] Step A5: Set all zero matrices within the rectangular area to 1.

[0013] Step A6: Finally, save the generated mask matrix as an npy format file to obtain the mask matrix.

[0014] Step A7: Perform a dot product between the mask matrix and the original image to obtain a rectangular image containing only the lesion area.

[0015] Step 3: Divide the original dataset into training and test sets in a 7:3 ratio.

[0016] Step 4: Use convolutional neural networks to extract lesion region features from the training set. For optimal results, ResNet is chosen as the image feature extractor for MG images, and a multi-scale attention mechanism network is chosen for DBT images. Fully connected layers serve as classifiers for their respective networks. Cross-entropy loss is used as the network's loss function. The Adam optimizer is used for parameter updates. The Adam optimizer naturally implements the compensated annealing process, i.e., it automatically changes the learning rate. AUC is used as the evaluation metric for the classification model.

[0017] Step 5: Using the classification training method in Step 4, train the DBT image network based on DBT images to obtain the optimal DBT image feature extractor and classifier.

[0018] Step 6: Using the classification training method in Step 4, train the MG image network based on the MG image to obtain the optimal MG image feature extractor and classifier.

[0019] Step 7: Use the feature extractor trained in Steps 5 and 6 to extract features from DBT and MG images.

[0020] Step 8: Input the MG image features into the autoencoder for training. The autoencoder consists of an encoder and a decoder, and the loss function of the autoencoder is...

[0021] loss AE =MSE(x,Dec(Enc(x)))

[0022] Where MSE is the mean squared error, Dec is the decoder network, Enc is the encoder network, and x is the feature of the MG. The parameters of the autoencoder network are updated using the Adam optimizer, and the MG image features are used by the encoder to generate latent variables z with higher expressive performance.

[0023] Step 9: Input the latent variable z and DBT image features into a generative adversarial network, where the generator's loss function is:

[0024]

[0025] The discriminator loss function is:

[0026]

[0027] In the formula, MAE is the mean absolute error, x f To generate DBT features, x r These are the true DBT features. The generator and discriminator learn adversarially to make the generated DBT feature distribution approximate the true DBT feature distribution.

[0028] Step 10: Repeat steps 8 and 9 until the parameters in the autoencoder and generative adversarial network converge and no longer update, and the network model training is complete.

[0029] Step 11: On the test set, test the benign and malignant classification performance of the generated DBT images.

[0030] The beneficial effects of this invention are as follows: To further improve the classification performance of MG images, this invention designs a cross-modal DBT image feature generation model. The model inputs MG and DBT image features extracted by a convolutional neural network, and utilizes a generative adversarial network to make the MG features more similar to the DBT image features. In the application stage, the method only inputs MG images with lower information content, extracts image features, and generates DBT image features with higher information content through the cross-modal generation model for benign / malignant classification. Compared with traditional benign / malignant diagnosis methods based on MG images, this method exhibits better classification performance. This research is of great significance for using MG images for early assisted diagnosis. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a structural framework diagram of the present invention.

[0033] Figure 2 This is a flowchart of the network structure during the training phase of this invention.

[0034] Figure 3 This is a flowchart of the network structure during the testing phase of this invention.

[0035] Figure 4 The ROC curve for benign and malignant diagnosis in this invention is shown. CNN represents the neural network corresponding to the image, and GAN represents the feature generation model whose input is the corresponding image. Detailed Implementation

[0036] To make the objectives, features, and advantages of this invention more apparent and understandable, the embodiments of this invention are clearly and completely described below with reference to the specific implementation methods illustrated in the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort, such as those that only change the application without altering the implementation of the basic principles involved in the claims, fall within the scope of protection of this invention.

[0037] This invention provides a method for generating cross-modal breast tomographic image features, such as... Figure 1 As shown, the cross-modal breast tomographic image feature generation method consists of four modules: data processing module 1, feature extraction module 2, feature generation module 3, and image classification module 4.

[0038] Data processing module 1 performs data preprocessing on MG and DBT images respectively. It extracts lesion regions from the initial images based on doctor annotations and rotates, flips vertically and horizontally at specified angles (90°, 180°, 270°). It also expands the dataset to ensure the data distribution is not excessively skewed.

[0039] Feature extraction module 2, taking benign / malignant classification as an example, uses convolutional neural networks (CNNs) that perform well in benign / malignant classification to extract features from preprocessed MG and DBT images. The final feature extraction network is obtained by retaining the CNN portion and removing the fully connected layers.

[0040] Feature generation module 3 uses a generative adversarial network to train DBT feature generation. During the training process, MG features, DBT features and labels are input simultaneously. The generator generates DBT image features based on MG image features. The discriminator distinguishes between the generated features and the real DBT image features. If the features generated by the generator can confuse the discriminator, then the generated features are considered to be of high quality.

[0041] Image classification module 4, taking benign and malignant as an example, classifies by generating features and uses the AUC index for classification evaluation.

[0042] like Figure 2As shown, the training process of this invention is as follows: the cross-modal DBT image feature generation model consists of six parts, where each structural unit and its function are as follows:

[0043] (1)CNN1: A feature extractor based on convolutional neural networks, used to extract features from MG images;

[0044] (2) CNN2: A feature extractor based on convolutional neural networks, used to extract DBT image features;

[0045] (3) Enc: Encoder, which encodes the features of the input MG image and extracts high-dimensional features;

[0046] (4) Dec: Decoder, which decodes the features output by Enc and reconstructs the features of the MG image;

[0047] (5) G: Generator, which generates new DBT features x using the output features of Enc. f ;

[0048] (6) D: Discriminator, which analyzes the true DBT features x of the input. r and the generated DBT features x f Perform the discrimination and output the discrimination type.

[0049] In image feature extraction, deep learning models based on convolutional networks (CNNs) offer more accurate and robust feature representations compared to traditional methods. CNN1 and CNN2 represent the best convolutional neural networks for benign and malignant diagnosis of MG and DBT images, respectively. Removing the fully connected layers from these networks allows them to be used as image feature extractors.

[0050] Enc and Dec are two structures in an autoencoder. An autoencoder uses MG (mass image) features as its learning target, performing representation learning on these features. Enc encodes the input MG image features and extracts highly expressive latent variables z. Because there is significant noise between the MG image features of different patients, Enc reduces the specificity of MG features during dimensionality reduction, thus the latent variables z contain rich MG feature information and are insensitive to noise. Dec reconstructs the MG feature x using the latent variables z. The training objective of the autoencoder is:

[0051]

[0052] G and D form a generative adversarial network. Generator G uses the latent variable z to generate new DBT image features x.faker Discriminator D receives real DBT image feature information x real The generator then scores the generated features. G will generate DBT image features x that are as realistic as possible. faker D then distinguishes features for classification, generating image features x during the adversarial process. f It will become increasingly closer to the true image characteristics. The latent variable z contains rich case information from MG images. Although the imaging principle parameters are different, the two modalities of images are consistent in reflecting lesion information.

[0053] In the model, the latent variable z is used by Dec and G respectively. In Dec, the latent variable z is required to have a feature distribution similar to that of MG features. In the generator, since the generated model produces pseudo-DBT features, the latent variable z is required to have more DBT feature information. In summary, the latent variable z will contain as much of the same information as possible between MG and DBT image features.

[0054] The training process consists of three phases:

[0055] (1) Autoencoder training;

[0056] (2) Generator and encoder training;

[0057] (3) Discriminator training.

[0058] Modeling and design were carried out for each of the three stages.

[0059] In autoencoder training, the loss function describes the difference between the encoder input features and the decoder output features. An optimizer is used to refine the loss function, making the decoder output features closer to the features of the real MG image. The autoencoder loss function, obtained by obtaining the highly expressive latent variable z, is as follows:

[0060] loss AE =MSE(x, Dec(Enc(x)))

[0061] In generator and encoder training, the LSGAN design of this invention uses the following loss functions for the generator and encoder:

[0062]

[0063] In the formula x f x represents the features generated by the generator. r For DBT image features, mean absolute error is used to narrow the Euclidean distance between features, ensuring consistency in category attributes. This prevents situations where a true feature represents a malignant lesion, while the generated feature represents a benign lesion.

[0064] In the training of the discriminator, the discriminator loss function based on LSGAN is designed as shown in the equation.

[0065]

[0066] like Figure 3 The diagram illustrates the application process of this invention. Only MG images are input, and a trained encoder and generator are used to generate new features, taking benign / malignant transformation as an example. These new features are then used for breast cancer benign / malignant diagnosis. A convolutional network is used as the feature extractor for MG images to extract features x. Subsequently, the encoder Enc, trained using a cross-modal DBT image feature generation model, is used to extract features from x, obtaining latent variables z that share common features with both MG and DBT images. The generator then uses these latent variables z to generate pseudo-DBT image features x. faker Finally, a pre-trained DBT image feature classifier is used to classify pseudo-DBT images to obtain benign or malignant diagnostic results.

[0067] like Figure 4 As shown in the figure, CNN represents the neural network corresponding to the image, and GAN represents the feature generation model that takes the corresponding image as input. Comparing the three sets of experiments, DBT image outperforms the other two sets of experiments in all indicators, exhibiting the best classification performance. The GAN model inputs DBT image features into MG image features through generator loss and continuously optimizes the MG image features to have the prior distribution of DBT image features. Compared with MG features alone, it is easier to diagnose benign or malignant conditions, resulting in better diagnostic performance.

[0068] Bootstrap sampling is an important statistical diagnostic method. Its basic idea is to infer data samples by re-sampling from the sample. A significance test was performed on the AUC values ​​obtained from the generative model and the convolutional model of MG. The experiment showed that the AUC value of the generative model was significantly different from that of the convolutional model of MG (P-value = 0.002). When inputting MG images, the generative model (AUC = 0.878) outperformed the convolutional model (AUC = 0.844) in the benign / malignant classification task. Since the classifiers for features are the same in both models, it can be inferred that the generated DBT image features have a significant improvement in benign / malignant classification performance compared to MG image features. Comparing the AUC values ​​of the DBT convolutional model and the generative model, the results showed no significant difference in benign / malignant classification performance between the real DBT image features and the generated DBT image features (P-value = 0.257 > 0.005), indicating that the generated DBT features have higher classification value.

[0069] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A method for generating cross-modal breast tomographic image features, characterized in that, Includes the following steps: Step 1: Construct a dataset of breast cancer images, which includes DBT images, MG images, and benign / malignant labels for breast cancer. Divide the dataset into a training set and a test set, and preprocess the dataset. Step 2: Use a convolutional neural network to extract features of the lesion region in the training set; Step 3: Train the convolutional neural network using the training set to obtain the trained feature extractor; Step 4: Use the feature extractor trained in Step 3 to extract features from the DBT and MG images; Step 5: Input the MG image features into the autoencoder for training. The autoencoder consists of an encoder and a decoder, and the loss function of the autoencoder is... loss AE =MSE(x,Dec(Enc(x))) Where MSE is mean squared error, Dec is decoder network, Enc is encoder network, x is MG feature, the parameters of autoencoder network are updated using Adam optimizer, and MG image features are encoded by encoder to generate latent variable z with higher expressive performance. Step 6: Input the latent variable z and DBT image features into a generative adversarial network (GAN), where the loss function of the generator in the GAN is: The discriminant loss function of a generative adversarial network is: In the formula, MAE is the mean absolute error, x f To generate DBT features, x r To obtain true DBT features, the generator and discriminator use adversarial learning to make the generated DBT feature distribution closer to the true DBT feature distribution. Step 7: Repeat steps 5 and 6 until the parameters in the autoencoder and generative adversarial network converge and no longer update, and the network model training is complete. Step 8: Use the test set to test the benign and malignant classification performance of the generated DBT images.

2. The method for generating cross-modal breast tomographic image features according to claim 1, characterized in that, The dataset is divided into training and testing sets in a 7:3 ratio.

3. The method for generating cross-modal breast tomographic image features according to claim 1, characterized in that, In step 1, the dataset preprocessing involves extracting lesion areas from breast cancer images using doctor-annotated information and performing routine data augmentation.

4. The method for generating cross-modal breast tomographic image features according to claim 3, characterized in that, The steps for extracting the lesion area are as follows: Step A1: Generate an all-zero matrix that is exactly the same size and shape as the original image; Step A2: Set all zero matrices within the lesion area to 1 according to the annotation information; Step A3: Use a rectangle to find the boundary and find the longest side along the x-axis or y-axis as the side length of the square; Step A4: Adjust the entire rectangle along its short side to ensure that the center of the square mask remains the same as the center of the original rectangular mask; Step A5: Set all zero matrices within the rectangular area to 1; Step A6: Finally, save the generated mask matrix as an npy format file to obtain the mask matrix; Step A7: Perform a dot product between the mask matrix and the original image to obtain a rectangular image containing only the lesion area.

5. The method for generating cross-modal breast tomographic image features according to claim 2, characterized in that, For the MG images, ResNet is selected as the image feature extractor, and for the DBT images, a multi-scale attention mechanism network is selected as the image feature extractor. Fully connected layers are used as classifiers for both ResNet and the multi-scale attention mechanism network. Cross-entropy loss is used as the loss function for both ResNet and the multi-scale attention mechanism network. The Adam optimizer is used for parameter updates, and AUC is used as the evaluation metric for the classification model.

6. The method for generating cross-modal breast tomographic image features according to claim 5, characterized in that, The optimal DBT image feature extractor and classifier are obtained through training with DBT images; the optimal MG image feature extractor and classifier are obtained through training with MG images.

Images