A breast tumor segmentation method based on multi-modal ultrasound images

Abstract

A breast tumor segmentation method based on multi-modal ultrasound images is provided to solve the problems that the existing breast ultrasound image segmentation technology is prone to misidentifying artifacts and low echo regions as tumors and the tumor boundary segmentation precision is low, comprising the following steps: 1, acquiring paired B-mode ultrasound images and strain elasticity images, and constructing a multi-modal breast ultrasound image dataset; 2, building a multi-modal image segmentation model algorithm based on feature correction alignment and boundary uncertainty enhancement; 3, training the multi-modal image segmentation model based on feature correction alignment and boundary uncertainty enhancement in a five-fold cross-validation manner, and saving the optimal model parameters; 4, loading the optimal model parameters, performing inference prediction on the images in the validation set, calculating the evaluation indexes on the validation set, and comparing with the current other advanced algorithms. The present application can effectively improve the segmentation precision of breast tumors in ultrasound images and improve the accuracy of network prediction. The present application belongs to the field of computer vision, deep learning and medical image segmentation.

Description

Technical Field

[0001] This invention relates to a breast tumor segmentation algorithm based on multimodal ultrasound images, belonging to the fields of computer vision, deep learning, and medical image segmentation. Background Technology

[0002] Breast cancer is one of the most common cancers, posing a significant threat to women's health. Timely detection and treatment are crucial for improving the survival rate of breast cancer patients. Breast ultrasound, as an early screening tool for breast cancer, is widely used due to its affordability, non-invasiveness, and lack of ionizing radiation. However, accurately delineating the tumor outline from B-mode ultrasound images is extremely challenging and time-consuming due to severe speckle noise, artifacts, and lateral echoes, as well as the often blurred tumor boundaries. To address these issues, numerous automated breast ultrasound image segmentation methods have been proposed to assist physicians in diagnosis and save time.

[0003] In recent years, deep learning-based methods have been widely used in breast ultrasound image segmentation, with the vast majority of these methods based on single-modal B-mode ultrasound images. However, artifacts and hypoechoic regions present in B-mode ultrasound can interfere with tumor detection, making it difficult for single-modal B-mode ultrasound image-based segmentation methods to achieve satisfactory results. Considering this issue, ultrasound physicians in clinical practice often follow the BI-RADS standard to employ multimodal image-assisted diagnosis, such as B-mode ultrasound and strain elastography, using multimodal images to delineate breast tumor contours and make clinical decisions.

[0004] Strain elastography, an imaging technique for assessing the stiffness of breast tumors and surrounding tissues, has become an important component of breast ultrasound examinations. Generally, the stiffness of breast tumors differs significantly from that of surrounding normal breast tissue. By displaying these stiffness differences between different tissues, strain elastography can provide supplementary information about the tumor's location and shape. In clinical ultrasound examinations, sonographers often combine B-mode ultrasound with strain elastography to determine the morphological boundaries of breast tumors.

[0005] A few researchers have conducted studies on breast tumor segmentation algorithms based on B-mode ultrasound and strain elastography, but these are all based on traditional machine learning algorithms, requiring manual adjustment of parameters and thresholds. They fail to fully utilize the complementary features of the two modalities and ignore the potential mutual interference of inherent noise in the two modalities. Meanwhile, severe speckle noise in B-mode ultrasound and the irregular shape of malignant tumors also blur the tumor boundaries. Summary of the Invention

[0006] To address the problems of existing breast ultrasound image segmentation techniques that easily misidentify artifacts and hypoechoic regions as tumors and have low accuracy in tumor boundary segmentation, this invention proposes a breast tumor segmentation method based on multimodal ultrasound images.

[0007] The technical solution adopted by the present invention to solve the above problems is as follows:

[0008] This invention includes the following steps:

[0009] Step 1: Obtain paired B-mode ultrasound images and strain elastic images to construct a multimodal breast ultrasound image dataset:

[0010] Step 2: Build a multimodal image segmentation model algorithm based on feature correction alignment and boundary uncertainty enhancement;

[0011] Step 3: Train a multimodal image segmentation model based on feature correction alignment and boundary uncertainty enhancement using five-fold cross-validation, and save the parameters of the best-performing model.

[0012] Step 4: Load the optimal model parameters, perform inference prediction on the images in the validation set, calculate the evaluation index on the validation set, and compare it with other current advanced algorithms.

[0013] Furthermore, step 1 includes:

[0014] Step 1.1 Acquire paired B-mode ultrasound images and strain elastic images;

[0015] Step 1.2 An ultrasound physician with more than five years of experience annotates the tumor outline in the B-mode ultrasound image and saves it as a binary image;

[0016] Step 1.3 divides the dataset into five folds.

[0017] Furthermore, step 2 includes:

[0018] Step 2.1 The model consists of two backbone networks, an attention-based feature correction and aggregation module, and a boundary uncertainty enhancement module. The two backbone networks are ResNet-34 networks pre-trained on ImageNet, extracting features at four different scales. The features at the four corresponding scales of the two modalities are input into the attention-based feature correction and aggregation module for feature fusion. Constraint supervision is applied to the upsampled segmentation results, preliminary segmentation results, and final segmentation results of the fused features output at different scales.

[0019] Step 2.2 The structure of the attention-based feature correction aggregation module is as follows: Figure 2As shown, this module consists of a feature correction section and a feature alignment section. The feature correction section uses a spatially guided attention module to extract effective spatial features from different modalities to guide the removal of noise and interference present in another modality. The feature aggregation section first uses a multi-scale feature mapping method to extract features of different scales from the two modalities and maps them to the same domain, then uses a self-attention mechanism to align the multi-scale features of the two modalities.

[0020] Step 2.3: After the features from different scales are fused by the attention-based feature correction and aggregation module, they are upsampled and concatenated along the channel dimension. This concatenation is then followed by a 3x3 convolution and a 1x1 convolution to obtain the initial segmentation result. This initial segmentation result, along with the previous one, is input into the boundary uncertainty enhancement module. Figure 3 As shown, the final segmentation result is obtained.

[0021] Step 2.4 Loss function L for network training total It consists of three parts, namely, deep supervision loss L aux Initial segmentation result loss L init and final segmentation loss L final The definition is as follows:

[0022]

[0023] L init =L Dice (Y init ,Y)

[0024] L final =L Dice (Y final ,Y)

[0025] L total =L final +L aux +λ1·L init

[0026] Where Y i N represents the prediction result obtained after feature upsampling from the output of each feature fusion module. layer The number of feature levels is 4, Y init Y represents the initial segmentation result of the output. final This represents the final segmentation result, where Y represents the segmentation label, and L represents the segmentation label. Dice L represents the Dice loss function. BCE This represents the binary cross-entropy loss function, λ1 = 0.2.

[0027] Furthermore, in step 3, four folds of the data are used as the training set, and the other fold is used as the validation set.

[0028] Furthermore, in step 4, five metrics—Dice coefficient, Jaccard index, recall, precision, and specificity—are used to compare the results with other advanced single-modal and multimodal methods.

[0029] The beneficial effects of this invention are as follows:

[0030] 1. This invention proposes using B-mode ultrasound images and strain elastic images for multimodal image segmentation of breast tumors, which can effectively reduce the false identification of artifacts and hypoechoic regions;

[0031] 2. The attention-based feature correction and aggregation module proposed in this invention can effectively extract complementary features from different modalities and achieve feature correction and multi-scale feature alignment, thereby realizing feature fusion;

[0032] 3. The boundary uncertainty enhancement module proposed in this invention can effectively enhance the determinism of boundary region segmentation results, and obtain more accurate tumor contours and edges;

[0033] 4. This invention uses deep supervision loss and multi-step segmentation loss, which can effectively constrain the edge contour of the tumor;

[0034] 5. The multimodal segmentation network based on B-mode ultrasound images and strain elastic images used in this invention can effectively improve the segmentation accuracy of breast tumors in ultrasound images and improve the accuracy of network prediction. Attached Figure Description

[0035] Figure 1 This is a structural diagram of the breast tumor segmentation model based on multimodal ultrasound images proposed in this invention;

[0036] Figure 2 This is a structural diagram of the attention-based feature correction aggregation module proposed in this invention;

[0037] Figure 3 This is a structural diagram of the boundary uncertainty enhancement module proposed in this invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the specific embodiments of this invention are described in further detail below:

[0039] The dataset used in this invention is a multimodal breast ultrasound image dataset acquired at the Second Affiliated Hospital of Harbin Medical University. This dataset contains 604 sets of B-mode ultrasound images and strain elastography images from 547 breast tumor patients. All ultrasound images were acquired by experienced sonographers using Siemens S2000 and Philips iU14 scanners. The MBUSI dataset includes 210 benign cases (215 image pairs) and 337 malignant cases (389 image pairs). Each case includes a B-mode ultrasound image and a corresponding strain elastography image. The marginal contours of the breast tumors were annotated by sonographers with over five years of experience.

[0040] This implementation method is achieved through the following steps:

[0041] Step 1: Obtain paired B-mode ultrasound images and strain elastic images to construct a multimodal breast ultrasound image dataset:

[0042] 1) Unify the acquired paired B-mode ultrasound and strain elastic images and their corresponding segmentation labels to the same size of 256*256.

[0043] 2) Divide the dataset into five equal folds. During each training session, use one fold as the validation set and the other four folds as the training set.

[0044] 3) Perform data augmentation on the training set by randomly flipping it horizontally or vertically, and randomly adding Gaussian noise.

[0045] Step 2: Construct a multimodal image segmentation model based on feature correction alignment and boundary uncertainty enhancement. The network structure is as follows: Figure 1 As shown, it specifically includes the following:

[0046] 1) The model proposed in this invention consists of three parts: two backbone branches, an attention-based feature correction and aggregation module, and a boundary uncertainty enhancement module. Each backbone branch is a ResNet-34 pre-trained on the ImageNet dataset. First, paired B-mode ultrasound images and strain elastic images are input into the two branch networks respectively, resulting in four stages of output feature layers. The sizes of the four feature layers are 64×64×64, 128×32×32, 256×16×16, and 512×8×8, respectively.

[0047] 2) Input the feature maps corresponding to B-mode ultrasound images and strain elastic images of the same size into the same attention-based feature correction aggregation module, with the structure as follows: Figure 2As shown, the two input feature layers each use a spatial attention mechanism to extract their spatial weights to correct another modality. Finally, the features of the two modalities are fused by concatenating the channel dimensions to obtain a multimodal attention vector I. A filtered feature layer is obtained using a sigmoid activation function and a multilayer perceptron, which is added to the original input feature layer to obtain the corrected feature layer, thus correcting the features of both modalities. Then, multi-scale feature mapping is performed on the two modalities respectively, and 1*1, 3*3, and 5*5 convolution operations are performed on the feature layers, with the results added to obtain Q, K, and V. A cross-attention mechanism is used to align the two modalities. Finally, the two attentions are processed through a softmax function to obtain corresponding weights, which are multiplied by the original modal features and added to obtain the final fused feature.

[0048] 3) Upsample the fused features output from the four attention-based feature correction and aggregation modules to the same size, perform convolution operations, and then calculate the loss L. aux Then, the four output feature layers are concatenated along the channel dimension to obtain a fused feature layer. The fused feature layer undergoes a 3x3 convolution and a 1x1 convolution to obtain the initial segmentation result. The loss L is calculated on the initial segmentation result. init The fused feature layer and the initial segmentation results are input into the boundary enhancement module, with the structure as follows: Figure 3 As shown.

[0049] 4) Reverse the initial segmentation result to obtain the reverse saliency map and boundary map. Concatenate the three feature maps along the channel dimension to obtain the uncertainty enhancement feature map, multiply it with the fusion feature layer obtained in the previous step, and then perform a convolution operation to obtain K and V. Perform a convolution operation on the fusion feature layer to obtain Q, perform a self-attention mechanism to calculate it, and add it to the initial segmentation result to obtain the final segmentation result. Calculate the loss L. final .

[0050] Step 3: Train a multimodal image segmentation model based on feature correction alignment and boundary uncertainty enhancement using five-fold cross-validation, and save the parameters of the best-performing model. Use four folds of data as the training set and the remaining fold as the validation set. The Dice loss function is used for the validation set, and the model that performs best on the validation set is saved. This process is repeated for each fold, using it as the validation set and saving the best-performing model for each fold.

[0051] Step four: Load the optimal model parameters, perform inference prediction on the images in the validation set, calculate the evaluation metrics on the validation set, and compare them with other current state-of-the-art algorithms. The five saved optimal model parameters are loaded and tested on the validation set data. The Dice coefficient, Jaccard index, recall, precision, and specificity are calculated. The average value of the metrics during the five-fold cross-validation process is calculated. The experimental results are shown in Table 1.

[0052] Table 1 compares the segmentation results of different methods on the MBUSI dataset.

[0053]

[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A method for segmenting breast tumors based on multimodal ultrasound images, characterized in that, Includes the following steps: Step 1: Obtain paired B-mode ultrasound images and strain elastic images to construct a multimodal breast ultrasound image dataset; Step 2: Build a multimodal image segmentation model based on feature correction alignment and boundary uncertainty enhancement; The model consists of two independent backbone branches, an attention-based feature correction and aggregation module, and a boundary uncertainty enhancement module. Among them, the two main branch networks are ResNet-34 networks pre-trained on the ImageNet dataset, used to extract features at four different scales from the B-mode ultrasound image and the strain elastic image, respectively. The attention-based feature correction and aggregation module is used to fuse features of the same scale from two modalities, and includes: The feature correction section utilizes the spatial guidance attention module to extract effective spatial features of different modalities to guide the removal of noise and interference present in another modality. In the feature alignment part, multi-scale feature mapping is performed on the two modal features after correction. The multi-scale feature mapping is achieved by using 1×1, 3×3 and 5×5 convolution kernels in parallel and adding them together. Then, a cross-attention mechanism is used to align and aggregate the mapped features to obtain fused features. The boundary uncertainty enhancement module includes: performing a reverse operation on the initial segmentation result to generate a reverse saliency map, and combining the boundary map with the initial segmentation result to generate an uncertainty enhancement feature map; fusing the feature map with the fusion feature output by the feature correction aggregation module, and calculating the final segmentation result through a self-attention mechanism; The model's total training loss function Losses due to deep supervision Initial segmentation loss and final segmentation loss It consists of three parts, and is defined as follows: L total =L final +L aux +λ1·L init in, This is the weighting coefficient, with a value of 0.2; Step 3: Train the multimodal image segmentation model using five-fold cross-validation and save the optimal model parameters; Step 4: Load the optimal model parameters, perform inference prediction on the validation set images, and calculate the evaluation index on the validation set.

2. The method for breast tumor segmentation based on multimodal ultrasound images according to claim 1, characterized in that, In step 1, paired B-mode ultrasound and strain elastic images are collected to construct a multimodal breast ultrasound image dataset, which is used to train the multimodal image segmentation model.

3. The method for breast tumor segmentation based on multimodal ultrasound images according to claim 1, characterized in that, In step 3, the model performance is evaluated using a five-fold cross-validation method. Each training session uses four folds of data as the training set and the remaining fold as the validation set, and saves the model parameters with the lowest loss on each fold of the validation set.

4. The method for breast tumor segmentation based on multimodal ultrasound images according to claim 1, characterized in that, In step 4, the segmentation performance evaluation metrics include the Dice coefficient, Jaccard index, recall, precision, and specificity.

Images