Mammary gland ultrasound image tumor classification network assisted by segmentation task

Abstract

The invention provides a dual-branch breast ultrasound image tumor classification network based on a TransUNet and a Transformer. As the internal region and the boundary region of the breast ultrasound image have a key effect on the benign and malignant interpretation of the tumor, the internal features and the boundary features of the tumor are respectively extracted through the proposed features extracted by combining the TransUNet and the ResNet of the double-branch structure, and the fusion of spatial information and semantic information is realized. Meanwhile, due to the fact that malignant tumors can influence peripheral tissues, a TConv module is provided to extract global features, and the global features are fused with local features to learn the relation between the tumors and the peripheral tissues. Experiments are carried out on a disclosed data set B and a data set BUSI, and experimental results show the effectiveness of the network.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, computer vision, and medical imaging, and proposes a tumor classification network for breast ultrasound images assisted by segmentation tasks. Background Technology

[0002] Breast tumors have various subtypes, and the progression from precancerous lesions to malignant cancer takes a long time. Early diagnosis of cancer can significantly improve treatment outcomes and reduce cancer mortality. Intelligent tumor classification in breast ultrasound images based on deep learning can effectively improve diagnostic efficiency, serving as an auxiliary diagnostic tool for doctors. Tumors in breast ultrasound images vary in size and shape, and benign and malignant tumors differ significantly in their boundary characteristics. Benign tumors are surrounded by a capsule, do not have internal calcifications or have coarse calcifications, have smooth and regular boundaries, do not invade surrounding normal tissue, exhibit an expansive growth pattern, and grow relatively slowly. In contrast, malignant tumors, compared to benign tumors, have lower internal echoes, irregular and blurred tissue boundaries, exhibit an invasive growth pattern, and grow relatively quickly. Therefore, when determining the benignity or malignancy of breast tumors, ultrasound physicians consider not only the size, internal structure, and tissue edges of the tumor but also the condition of surrounding tissues. However, the inherent nature of breast ultrasound imaging results in low contrast between the surrounding tissue and the tumor area, making it challenging for the model to extract boundary features. Furthermore, the benign or malignant nature of breast tumors depends not only on the boundary region but also on the condition of surrounding tissues. To avoid overtreatment due to misdiagnosing benign tumors as malignant tumors and missing the optimal treatment window due to misdiagnosing malignant tumors as benign tumors, accurate classification and improved classification accuracy are necessary. Breast tumor classification technology based on deep learning networks holds great potential, and research into intelligent classification networks for breast ultrasound lesions based on deep learning can effectively improve segmentation performance and diagnostic efficiency.

[0003] The innovation of this invention and its difference from the prior art

[0004] Existing breast ultrasound image classification networks mainly fall into two categories: machine learning-based and deep learning-based. Machine learning-based breast ultrasound classification networks involve two key steps: feature extraction and classifier design. However, the feature extraction stage relies on manually setting initial parameters, which significantly impacts the final classification result. Deep learning-based breast tumor classification networks, on the other hand, automatically extract features based on the inherent characteristics of the image, reducing the influence of human factors. Existing deep learning-based breast ultrasound image classification networks mostly classify feature maps based on classic backbone networks. However, due to the inherent properties of benign and malignant tumors, there are significant differences within the tumor itself, and malignant tumors exhibit invasiveness to surrounding tissues, necessitating consideration of global information in the breast ultrasound image. Existing networks only consider the internal region of the tumor, neglecting global tumor information. This invention addresses this issue by using a dual-branch module and...

[0005] The TConv module guides the network to learn local and global features in breast ultrasound images. Summary of the Invention

[0006] To address the issue of insufficient accuracy in classifying lesions using existing breast ultrasound images, this invention focuses on the characteristics of the mass region and global features in breast ultrasound images to achieve better classification performance.

[0007] The technical solution adopted in this invention is a breast ultrasound image tumor classification network assisted by segmentation task. This network takes the original ultrasound image as input and outputs the final image category after training. The network method includes two processes: building the DTNet model and training the model.

[0008] The network structure of this invention is shown below. Figure 3 The specific network steps are as follows:

[0009] Step 1: Construct the DTNet model

[0010] (1) Since the internal and boundary regions of breast ultrasound images play a crucial role in determining the benign or malignant nature of tumors, this invention proposes a dual-branch module based on TransUNet and ResNet to extract the internal and boundary features of tumors. By fusing the deep feature map with richer semantic features extracted by the TransUNet decoder stage with the shallow feature map with richer location information extracted by ResNet, the tumor can be better located and deep semantic information can be extracted.

[0011] like Figure 1 As shown.

[0012] (2) Furthermore, since malignant tumors in breast ultrasound images can affect surrounding tissues, this invention proposes a TConv module to extract global features. This module fuses the feature map containing global information extracted by the Transformer with the feature map containing local information obtained through convolution, allowing the network to learn the relationship between the tumor and surrounding tissues. For example... Figure 3 As shown.

[0013] Step 22: Model Training

[0014] (1) Divide the dataset into training set, validation set and test set (8:1:1) according to the original ratio of benign to malignant.

[0015] (2) Due to the limited number of medical image datasets, this work uses a transfer learning network. First, the model parameters pre-trained on the 1mageNet dataset are loaded, and then the breast ultrasound dataset is loaded to fine-tune the network.

[0016] (3) Images are input into the network in batches. One breast ultrasound image corresponds to one classification result and three segmentation results. The segmentation loss function is calculated based on the segmentation results and segmentation labels to guide the network to focus on the tumor region. The calculation formula is as follows, where y i ∈{0,1} represents the true segmentation label of a breast ultrasound image. This represents the network's prediction result, where N represents the number of pixels in the breast ultrasound image. This represents the loss between the three feature maps output by TransUNet and the corresponding ground truth labels. In the diagram, i∈{1,2,3} represents the three corresponding feature maps.

[0017]

[0018] The classification loss function is calculated based on the classification results and classification labels to guide the network to focus on relevant features. The calculation formula is as follows.

[0019]

[0020] The final loss function of the model consists of segmentation loss and classification loss, and the formula for calculating the loss function is shown below.

[0021]

[0022] This paper proposes a two-branch breast ultrasound image tumor classification network based on TransUNet. Since the internal and boundary regions of breast ultrasound images play a crucial role in determining the benign or malignant nature of tumors, a two-branch module based on TransUNet and ResNet is proposed to extract internal and boundary features of the tumor. The deep feature map, rich in semantic features, extracted by the TransUNet decoder stage, and the shallow feature map, rich in location information, extracted by ResNet, are fused to better locate the tumor and extract deep semantic information. Secondly, since malignant tumors in breast ultrasound images can affect surrounding tissues, this invention proposes a TConv module to extract global features. The feature map containing global information extracted by Transformer is fused with the feature map containing local information obtained through convolution, allowing the network to learn the relationship between the tumor and surrounding tissues. Attached Figure Description

[0023] Figure 1 This is the dual-branch module proposed in this invention;

[0024] Figure 2 The Transformer module used in this invention;

[0025] Figure 3 This is a diagram of the network model proposed in this invention;

[0026] Figure 4 This is a flowchart of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0028] The datasets used in this invention are the publicly available datasets B and BUSI. Dataset B was acquired in 2012 by the UDIAT Diagnostic Centre of the Parc Taulí Corporation, using a Siemens ACUSON Sequoia C512 system equipped with a 17L5HD linear array transducer at a frequency of 8.5MHz. This dataset contains a total of 163 breast ultrasound images, each with an average image size of 760*570 pixels. Of these images, 110 were diagnosed as benign tumors and 53 as malignant tumors. Dataset BUSI was acquired in 2018 from the Baheya Hospital for Early Detection and Treatment of Women's Cancer Center in Cairo, Egypt. The LOGIQ E9 and LOGIQ E9Agile ultrasound systems were primarily used during acquisition. This dataset contains images with an average size of 500*500 pixels, encompassing ultrasound images of benign tumors, malignant tumors, and normal breast tissue. In this work, we primarily used ultrasound images containing both benign and malignant tumors, including 487 images of benign tumors and 210 images of malignant tumors.

[0029] Step 1: Construct the DTNet model

[0030] (1) Since the internal and boundary regions of breast ultrasound images play a crucial role in determining the benign or malignant nature of tumors, this invention proposes a dual-branch module based on TransUNet and ResNet to extract the internal and boundary features of tumors. The deep feature map, rich in semantic features, extracted by the TransUNet decoder stage, and the shallow feature map, rich in location information, extracted by ResNet, are fused to better locate the tumor and extract deep semantic information. Branch 1 uses TransUNet to extract features of the tumor region and boundary region, removing the last layer of the TransUNet decoder to obtain feature maps T3, T2, and T1 of sizes B×256×28×28, B×128×56×56, and B×64×112×112, respectively (B represents Batchsize). The smaller feature maps aggregate information from large tumors, while the larger feature maps reflect information from small tumors. A convolutional module is added after each feature map to obtain feature maps of sizes B×128×28×28, B×64×56×56, and B×64×112×112, respectively. Branch 2 uses the first three layers of ResNet18 as the tumor feature extractor, obtaining feature maps R1, R2, and R3 of sizes B×64×112×112, B×64×56×56, and B×128×28×28, respectively. The feature maps obtained after multiple convolutions by the TransUNet encoder contain more semantic information; conversely, the feature maps obtained from the first three layers of ResNet, due to fewer convolutional blocks, retain more positional information. The feature maps obtained from TransUNet and ResNet are gradually fused to obtain the final feature map extracted by the backbone network, as shown below. Figure 1 As shown.

[0031] (2) Furthermore, since malignant tumors in breast ultrasound images can affect the surrounding tissues, this invention proposes a TConv module to extract global features. This module fuses the feature map containing global information extracted by the Transformer with the feature map containing local information obtained through convolution, allowing the network to learn the relationship between the tumor and surrounding tissues. The Transformer structure is as follows: Figure 2As shown. Considering the size of the network model, we used 6 Transformer modules in the TConv module, and in the subsequent forward fusion process, we fused the results of 2x and 4x downsampling using this Transformer module. First, the result G1 obtained from the dual-branch network is fed into the Transformer module to allow the network to learn the contextual information of the tumor. Then, the obtained feature map is added to G1 to obtain G2. Next, G2 is fed into the fourth layer of ResNet18, and the result of 2x downsampling by the Transformer is added to the obtained result to obtain G3. Then, G3 is fed into the fifth layer of ResNet18, and the result of 4x downsampling by the Transformer is added to the obtained result to obtain G4. Finally, G4 is converted into benign / malignant classification probability values ​​through global average pooling and a fully connected layer, as shown. Figure 3 As shown.

[0032] Step 2: Model Training

[0033] (1) Divide the dataset into training set, validation set and test set (8:1:1) according to the original ratio of benign to malignant.

[0034] (2) Due to the limited number of medical image datasets, this work uses a transfer learning network. First, the model parameters pre-trained on the !mageNet dataset are loaded, and then the breast ultrasound dataset is loaded to fine-tune the network.

[0035] (3) The DTNet proposed in this invention consists of two parts: a dual-branch module and a TConv module. The image first passes through the TransUNet branch in the dual-branch module, which extracts tumor region information, and the branch consisting of the first three layers of ResNet18, which extracts spatial information. Then, the results from the two branches are fused layer by layer to obtain a preliminary feature map. The TransUNet decoder generates feature maps of sizes 28×28, 56×56, and 112×112 layer by layer. For feature maps of different sizes, deep supervision techniques are added, downsampling the original labels by two, four, and eight times respectively, and calculating segmentation loss with the feature maps generated by TransUNet. This loss guides the feature maps generated in the hidden layers, making them more focused on the tumor region, such as... Figure 3 As shown, losses are calculated for T1 and L1, T2 and L2, and T3 and L3. The loss function combines the Binary Cross Entropy Loss (BCE) and the Dice loss function. Furthermore, the model outputs the final classification result, and the classification loss function is calculated using the classification result and the classification label. The final loss function of the model consists of both the segmentation loss and the classification loss.

[0036] The hyperparameters used during model training are shown in Table (1):

[0037] Table 1 Network Training Hyperparameters

[0038] Hyperparameters numerical values Training Algebra 100 Selected optimizer Adam optimizer Learning rate 0.0001 BatchSize 24

[0039] To verify the effectiveness and feasibility of the network of this invention, a comparative experiment with 5-fold cross-validation was conducted on dataset B and dataset BUSI. The experimental results are shown in Tables 2 and 3. The accuracy and F1 score of this network on dataset B are 89.41% and 83.52%, respectively, and the accuracy and F1 score on dataset BUSI are 89.7% and 82.89%, respectively, which proves that this network has better classification performance than other networks.

[0040] Table 2 shows the experimental comparison of the model on dataset B.

[0041] Method Accuracy% AUC% Precision% Recall% F1 Score% Inception V3 67.06±0.03 65.45±0.07 52.91±0.04 60.0±0.23 54.46±0.1 Alexnet 70.59±0.08 69.7±0.06 61.2±0.14 66.67±0.15 61.64±0.08 Vgg11 74.12±0.08 74.7±0.05 63.42±0.11 76.67±0.08 68.21±0.05 Vgg13 78.82±0.06 79.09±0.05 67.9±0.09 80.0±0.07 73.0±0.06 Vgg16 78.82±0.08 79.85±0.06 67.74±0.11 83.33±0.0 74.22±0.07 ResNet18 84.71±0.03 80.61±0.02 88.0±0.1 66.67±0.0 75.64±0.04 ResNet34 81.18±0.04 77.12±0.08 81.0±0.1 63.33±0.19 69.02±0.1 ResNet50 78.82±0.11 80.61±0.11 66.92±0.15 86.67±0.12 74.89±0.12 Our 89.41±0.02 86.52±0.03 93.33±0.08 76.67±0.08 83.52±0.04

[0042] Table 3 shows the experimental comparison of the models on the BUSI dataset.

[0043] Method Accuracy% AUC% Precision% Recall% F1 Score% Inception V3 74.24±0.06 71.46±0.05 61.27±0.1 63.81±0.11 61.21±0.06 Alexnet 82.73±0.02 76.67±0.02 81.35±0.06 60.0±0.05 68.78±0.03 Vgg11 80.91±0.03 75.59±0.04 74.99±0.08 60.95±0.06 67.04±0.05 Vgg13 80.3±0.03 76.16±0.05 71.25±0.06 64.76±0.12 67.27±0.07 Vgg16 80.91±0.02 74.83±0.02 76.96±0.06 58.1±0.06 65.87±0.04 ResNet18 84.24±0.02 79.3±0.02 81.72±0.06 65.71±0.02 72.72±0.03 ResNet34 85.15±0.03 80.98±0.02 82.03±0.09 69.52±0.02 75.03±0.04 ResNet50 82.73±0.02 77.68±0.02 79.28±0.09 63.81±0.04 70.27±0.03 Our 89.7±0.01 86.6±0.01 88.99±0.07 78.1±0.04 82.89±0.02

Claims

1. A breast ultrasound image tumor classification network under the assistance of segmentation tasks, characterized in that: The network comprises a model DTNet construction and training process. The specific steps are as follows: Step one: build the DTNet model (1) Since the internal and boundary regions of the breast ultrasound image play a key role in the interpretation of tumor benignity and malignancy, the present application proposes a double-branch module based on TransUNet and ResNet to extract the internal and boundary features of the tumor. The deep feature maps extracted by the TransUNet decoder stage are rich in semantic features, and the shallow feature maps extracted by the ResNet are rich in position information. The two kinds of feature maps are fused to better locate the tumor and extract deep semantic information. As shown in Figure 1. (2) In addition, since the malignant tumor in the breast ultrasound image will affect the surrounding tissue, the present application proposes a TConv module based on the above to extract global features. The feature maps extracted by the Transformer contain global information, and the feature maps obtained by convolution contain local information. The two kinds of feature maps are fused to make the network learn the relationship between the tumor and the surrounding tissue. As shown in Figure 2. Step two: model training (1) The data set is divided into training set, validation set and test set (8:1:1) according to the original benign and malignant ratio. (2) Since the medical image data set is too small, the method of transfer learning is adopted in this work. First, load the pre-trained model parameters on the ImageNet data set, and then load the breast ultrasound data set to fine-tune the network. (3) The image is input into the network in batches. One breast ultrasound image corresponds to one classification result and three segmentation results. The segmentation result and the segmentation label calculate the loss function, which guides the network to focus on the tumor area. The calculation formula is as follows. The classification result and the classification label calculate the loss function, which guides the network to focus on the relevant features. The calculation formula is as follows. The final loss function of the model is composed of the segmentation loss and the classification loss. The loss function calculation formula is shown as follows.

Images