Breast tumor segmentation method based on multi-modal ultrasonic image

Abstract

The invention relates to a breast tumor segmentation method based on a multi-modal ultrasonic image, which aims to solve the problems that artifacts and low-echo regions are easily wrongly recognized as tumors and the tumor boundary segmentation precision is low in the existing breast ultrasonic image segmentation technology, and comprises the following steps of: 1, acquiring a B-type ultrasonic image and a strain elastic image which are paired, and constructing a multi-modal breast ultrasonic image data set; 2, constructing a multi-modal image segmentation model algorithm based on feature correction alignment and boundary uncertainty enhancement; 3, training a multi-modal image segmentation model based on feature correction alignment and boundary uncertainty enhancement by adopting a five-fold cross validation mode, and storing model parameters with optimal performance; and 4, loading optimal model parameters, performing reasoning prediction on images in the verification set, calculating evaluation indexes on the verification set, and comparing the evaluation indexes with other current advanced algorithms. The method can effectively improve the segmentation precision of the breast tumor in the ultrasonic image, and improves the accuracy of network prediction. The invention belongs to the field of computer vision, deep learning and medical image segmentation.

Description

Technical Field

[0001] The present invention relates to a breast tumor segmentation algorithm based on multimodal ultrasound images, and belongs to the fields of computer vision, deep learning and medical image segmentation. Background Art

[0002] Breast cancer is one of the most common cancers and poses a major threat to women's health. Timely detection and treatment are crucial to improving the survival rate of breast cancer patients. Breast ultrasound, as a screening tool for early breast cancer, is widely used due to its reasonable price, non-invasiveness and non-ionizing radiation. However, since B-mode ultrasound images have severe speckle noise, artifacts and lateral echoes, and the tumor boundaries are very blurred, it is extremely challenging and time-consuming to accurately depict the contour of the tumor from B-mode ultrasound images. In order to solve these problems, a large number of automatic segmentation methods for breast ultrasound images have been proposed to assist doctors in diagnosis and save time.

[0003] In recent years, deep learning-based methods have been widely used in the field of breast ultrasound image segmentation. Most of these methods are based on single-mode B-mode ultrasound images. However, the artifacts and low-echo areas in B-mode ultrasound may interfere with tumor detection, making it difficult for segmentation methods based on single-mode B-mode ultrasound images to obtain satisfactory segmentation results. Considering this problem, ultrasound physicians often follow the BI-RADS standard in clinical practice to use multimodal images to assist diagnosis, such as B-mode ultrasound and strain elasticity images, to outline breast tumor contours and make clinical decisions.

[0004] Strain elastic imaging is an imaging technique for evaluating the hardness of breast tumors and surrounding tissues, and has become an important part of breast ultrasound examinations. Generally speaking, the hardness of breast tumors is often significantly different from that of surrounding normal breast tissue. Strain elastic imaging can provide supplementary information on the location and shape of tumors by showing the difference in hardness between different tissues. In clinical ultrasound examinations, ultrasound physicians often combine B-mode ultrasound with strain elastic imaging to determine the morphological boundaries of breast tumors.

[0005] A few researchers have conducted research on breast tumor segmentation algorithms based on B-mode ultrasound and strain elastic images, but they are all based on traditional machine learning algorithms, which require manual adjustment of parameters and thresholds, and cannot fully utilize the complementary features of the two modal images, ignoring that the inherent noise in the two modal images may affect each other. At the same time, the severe speckle noise in B-mode ultrasound and the irregular shape of malignant tumors also make the tumor boundaries unclear. Summary of the invention

[0006] In order to solve the problems that the existing breast ultrasound image segmentation technology easily misidentifies artifacts and low-echo areas as tumors and has low tumor boundary segmentation accuracy, the present invention further proposes a breast tumor segmentation method based on multimodal ultrasound images.

[0007] The technical solution adopted by the present invention to solve the above problems is:

[0008] The present invention comprises the following steps:

[0009] Step 1: Obtain paired B-mode ultrasound images and strain elasticity images to construct a multimodal breast ultrasound image dataset:

[0010] Step 2: Build a multimodal image segmentation model algorithm based on feature correction alignment and boundary uncertainty enhancement;

[0011] Step 3: Use a five-fold cross-validation method to train a multimodal image segmentation model based on feature correction alignment and boundary uncertainty enhancement, and save the model parameters with the best performance.

[0012] Step 4: Load the optimal model parameters, perform inference and prediction on the images in the validation set, calculate the evaluation indicators on the validation set, and compare them with other current advanced algorithms.

[0013] Furthermore, step 1 includes:

[0014] Step 1.1 Acquire paired B-mode ultrasound images and strain elasticity images;

[0015] Step 1.2 An ultrasound physician with more than five years of experience annotates the tumor contour in the B-mode ultrasound image and saves it as a binary image;

[0016] Step 1.3 divides the dataset into five folds.

[0017] Furthermore, step 2 includes:

[0018] Step 2.1 The model consists of two main branch networks, an attention-based feature correction aggregation module, and a boundary uncertainty enhancement module. The two main networks are ResNet-34 networks pre-trained on ImageNet, which extract features of four different scales. The features of the four corresponding scales of the two modalities are jointly input into the attention-based feature correction aggregation module for feature fusion. Constrained supervision is performed on the segmentation results, preliminary segmentation results, and final segmentation results obtained by upsampling the fused features output at different scales.

[0019] Step 2.2: The structure of the attention-based feature correction aggregation module is as follows: Figure 2As shown in Figure 2. This module consists of a feature correction part and a feature alignment part. The feature correction part uses the spatial guided attention module to extract effective spatial features of different modalities to guide the removal of noise and interference in another modality. The feature aggregation part first uses the multi-scale feature mapping method to extract different scale features of the two modalities, maps them to the same domain, and uses the self-attention mechanism to align the multi-scale features of the two modalities.

[0020] Step 2.3: After being fused by the attention-based feature correction aggregation module, the features of different scales are output and spliced ​​in the channel dimension after upsampling. After a 3*3 convolution and a 1*1 convolution, the initial segmentation result is obtained, which is input into the boundary uncertainty enhancement module together with the initial segmentation result. Figure 3 As shown, the final segmentation result is obtained.

[0021] Step 2.4 Network training loss function L total It consists of three parts: deep supervision loss L aux , initial segmentation result loss L init and the final segmentation loss L final , defined as follows:

[0022]

[0023] L init =L Dice (Y init ,Y)

[0024] L final =L Dice (Y final ,Y)

[0025] L total =L final +L aux +λ1·L init

[0026] where Y i Represents the prediction result obtained after upsampling the features output by each feature fusion module, N layer represents the number of feature levels is 4, Y init Represents the initial segmentation result of the output, Y final represents the final segmentation result of the output, Y represents the segmentation label, L Dice represents the Dice loss function, L BCE represents the binary cross entropy loss function, λ1=0.2.

[0027] Furthermore, in step 3, four folds of the data are used as training sets, and the other fold is used as a validation set.

[0028] Furthermore, in step 4, five indicators including Dice coefficient, Jaccard index, recall, precision and specificity are used to compare with other advanced single-modal and multimodal methods.

[0029] The beneficial effects of the present invention are:

[0030] 1. The present invention proposes to use B-mode ultrasound images and strain elasticity images for breast tumor segmentation, which can effectively reduce the misidentification of artifacts and low-echo areas;

[0031] 2. The attention-based feature correction aggregation module proposed in the present invention can effectively extract complementary features of different modalities and realize feature correction and multi-scale feature alignment to achieve feature fusion;

[0032] 3. The boundary uncertainty enhancement module proposed in the present invention can effectively enhance the certainty of the boundary area segmentation results and obtain more accurate tumor contours and edges;

[0033] 4. The present invention uses deep supervision loss and multi-step segmentation loss to effectively constrain the edge contour of the tumor;

[0034] 5. The multimodal segmentation network based on B-type ultrasound images and strain elasticity images used in the present invention can effectively improve the segmentation accuracy of breast tumors in ultrasound images and improve the accuracy of network prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a structural diagram of the breast tumor segmentation model based on multimodal ultrasound images proposed by the present invention;

[0036] Figure 2 This is a structural diagram of the attention-based feature correction aggregation module proposed in the present invention;

[0037] Figure 3 This is a structural diagram of the boundary uncertainty enhancement module proposed in the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention clearer, the specific embodiments of the present invention are further described in detail below:

[0039] The data set used in the present invention is a multimodal breast ultrasound image data set collected in the Second Affiliated Hospital of Harbin Medical University. The data set contains 604 groups of B-type ultrasound images and strain elastic images from 547 breast tumor patients. All ultrasound images are collected by experienced ultrasound physicians using Siemens S2000 and Philips iU14 scanners. The MBUSI data set includes 210 benign cases (215 groups of image pairs) and 337 malignant cases (389 groups of image pairs). Each case contains a B-type ultrasound image and a corresponding strain elastic image. The edge contours of breast tumors are annotated by ultrasound technicians with more than five years of experience.

[0040] This implementation is achieved through the following steps:

[0041] Step 1: Obtain paired B-mode ultrasound images and strain elasticity images to construct a multimodal breast ultrasound image dataset:

[0042] 1) The collected paired B-mode ultrasound and strain elasticity images and the corresponding segmentation labels are unified to the same size of 256*256.

[0043] 2) Divide the dataset into five equal folds. In each training, one fold is used as the validation set and the other four folds are used as the training set.

[0044] 3) Perform data enhancement methods such as random horizontal flipping, vertical flipping, and randomly adding Gaussian noise to the training set.

[0045] Step 2: Build a multimodal image segmentation model based on feature correction alignment and boundary uncertainty enhancement. The network structure is as follows: Figure 1 As shown, specifically including the following:

[0046] 1) The model proposed in this paper consists of two main branch networks, an attention-based feature correction aggregation module, and a boundary uncertainty enhancement module. Each main branch network is a ResNet-34 pre-trained on the ImageNet dataset. First, the paired B-mode ultrasound images and strain elasticity images are input into the two branch networks respectively, and the output feature layers of the four stages are obtained respectively. The sizes of the four feature layers are 64×64×64, 128×32×32, 256×16×16, and 512×8×8.

[0047] 2) The feature map corresponding to the B-mode ultrasound image and the feature map of the strain elastic image of the same size are input into the same attention-based feature correction aggregation module, the structure of which is as follows: Figure 2As shown. The two input feature layers use the spatial attention mechanism to extract their spatial weights to correct the other modality, and finally fuse the two modal features by splicing the channel dimension to obtain the multimodal attention vector I. The sigmoid activation function and multi-layer perceptron are used to obtain the filtered feature layer, which is added to the original input feature layer to obtain the corrected feature layer to achieve the correction of the two modal features. Then, multi-scale feature mapping is performed on the two modal features, and 1*1, 3*3, and 5*5 convolution operations are performed on the feature layers respectively, and the results are added to obtain Q, K, and V. The cross-attention mechanism is used to align the two modalities. Finally, the two attentions are passed through the softmax function to obtain the corresponding weights, multiplied by the original modal features, and added to obtain the final fused features.

[0048] 3) Upsample the fused features output by the four attention-based feature correction aggregation modules to the same size, perform convolution operations, and then output the calculated loss L aux , and then concatenate the four feature layers output in the channel dimension to obtain the fused feature layer. The fused feature layer undergoes a 3*3 convolution and a 1*1 convolution to obtain the initial segmentation result, and the loss L is calculated for the initial segmentation result init , the fusion feature layer and the initial segmentation result are input into the boundary enhancement module together, the structure is as follows Figure 3 shown.

[0049] 4) Perform a reverse operation on the initial segmentation result and calculate the reverse saliency map and boundary map. Concatenate the three feature maps in the channel dimension to obtain the uncertainty enhancement feature map, multiply it with the fusion feature layer obtained in the previous step, and then obtain K and V through convolution operation. Perform a convolution operation on the fusion feature layer to obtain Q, calculate it through the self-attention mechanism and add it to the initial segmentation result to obtain the final segmentation result, and calculate the loss L final .

[0050] Step 3: Use five-fold cross validation to train a multimodal image segmentation model based on feature correction alignment and boundary uncertainty enhancement, and save the best model parameters. Use four folds of data as training sets and the other fold as validation sets. The validation set loss function is the Dice loss function, and the model with the best performance on the validation set is saved. Use each fold as a validation set in turn, and save the best model on each fold of data.

[0051] Step 4: Load the optimal model parameters, perform inference prediction on the images in the validation set, calculate the evaluation indicators on the validation set, and compare them with other current advanced algorithms. Load the five saved optimal model parameters and perform inference tests on the validation set data, calculate the Dice coefficient, Jaccard index, recall rate, precision rate and specificity indicators, and calculate the average value of the indicators in the five-fold cross validation process. The experimental results are shown in Table 1.

[0052] Table 1 Comparison of segmentation results of different methods on the MBUSI dataset

[0053]

[0054] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A breast tumor segmentation method based on multimodal ultrasound images, characterized in that: The method comprises the following steps: Step 1, obtaining paired B-mode ultrasound images and strain elasticity images, and constructing a multimodal breast ultrasound image dataset; Step 2: Build a multimodal image segmentation model algorithm based on feature correction alignment and boundary uncertainty enhancement; Step 3: Use a five-fold cross validation method to train a multimodal image segmentation model based on feature correction alignment and boundary uncertainty enhancement, and save the best performing model parameters; Step 4: Load the optimal model parameters, perform inference and prediction on the images in the validation set, and calculate the evaluation indicators on the validation set.

2. The method for segmenting breast tumors based on multimodal ultrasound images according to claim 1, characterized in that: In the step 1, paired B-mode ultrasound and strain elasticity images are collected to construct a multimodal breast ultrasound image dataset for training the multimodal image segmentation model based on feature correction alignment and boundary uncertainty enhancement proposed in the present invention.

3. The method for segmenting breast tumors based on multimodal ultrasound images according to claim 1, characterized in that: The model described in step 2 includes two trunk branch networks, an attention-based feature correction aggregation module, and a boundary uncertainty enhancement module. The segmentation model algorithm includes the following: 2.1 In the attention-based feature correction and aggregation module, spatial attention is used to learn different modal features and perform feature correction, and multi-scale feature mapping and cross-attention mechanism are used to achieve feature alignment and aggregation; 2.2 In the boundary uncertainty enhancement module, the inverse saliency map and boundary map are used to perform uncertainty enhancement, and the segmentation loss is calculated for the initial segmentation result; 2.3 Calculate the deep supervision loss after upsampling the fused features output by the attention-based feature correction aggregation module.

4. The method for segmenting breast tumors based on multimodal ultrasound images according to claim 1, characterized in that: In step 3, four folds of data are used as training sets, and the other fold is used as a validation set.

5. The method for segmenting breast tumors based on multimodal ultrasound images according to claim 1, characterized in that: In step 4, five indicators, namely, Dice coefficient, Jaccard index, recall rate, precision rate and specificity, are used to compare with single-modality and multi-modality methods.

Images