A breast ultrasound mass segmentation method based on Transformer and multi-level feature aggregation

By using a Transformer-based multi-level feature aggregation method, the segmentation difficulties caused by noise and artifacts in breast ultrasound images were solved, achieving high-precision segmentation of breast masses and improving the accuracy and robustness of the segmentation model.

CN117541600BActive Publication Date: 2026-11-17CHINA THREE GORGES UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202311275327.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-28
Publication Date
2026-11-17
Estimated Expiration
2043-09-28

AI Technical Summary

Technical Problem

Existing technologies suffer from noise, artifacts, and low contrast in breast ultrasound images, resulting in unsatisfactory performance of traditional segmentation methods, especially in cases where the boundaries of breast masses are blurred and difficult to segment accurately.

Method used

Feature extraction is performed using a Transformer-based MiT encoder, combined with a multi-level feature aggregation method, including a MiT encoding layer, a multi-branch receptive field module, a permutation attention module, and an aggregation module. Through the fusion of multi-level feature maps and the capture of contextual information, accurate segmentation of breast lumps is achieved.

Benefits of technology

It improves the accuracy and robustness of breast mass segmentation, with a Dice coefficient of 85.14%, an IoU value of 77.54%, a Recall value of 86.80%, a Precision value of 85.05%, a Specificity value of 98.02%, and an Accuracy value of 97.1%, surpassing existing methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117541600B_ABST
    Figure CN117541600B_ABST
Patent Text Reader

Abstract

A method for breast ultrasound mass segmentation based on Transformer and multi-level feature aggregation includes the following steps: Step 1: Extract features from the breast ultrasound image and output a 4-layer feature map; Step 2: Input the output 4-layer feature map into the multi-branch receptive field (RBF) module of the decoding part; Step 3: Use the permutation attention module (SAM) to capture important information from the 4-layer feature map with extensive contextual information and output the feature map; Step 4: Use the aggregation module to fuse the output feature map S3 and feature map S4 to obtain feature map A1; use the aggregation module to fuse feature map A1 and feature map S2 to obtain feature map A2; use the aggregation module to fuse feature map A2 and feature map S1 to obtain feature map A3. This invention can solve the problems of noise in breast ultrasound images, low contrast between breast tissue and lesions, possible artifacts or pseudo-structures leading to missegmentation, and discontinuous segmentation in breast ultrasound mass segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of deep learning in medical image segmentation, specifically to a method for segmenting breast ultrasound masses. Background Technology

[0002] In existing technologies, due to the irregular shape and size of breast ultrasound masses, the presence of noise and artifacts in ultrasound images, unclear boundaries, and low contrast between the mass and surrounding tissues, traditional segmentation methods for breast masses mainly rely on information such as the edge contour of the target area, region, and threshold. Therefore, the traditional segmentation methods are not ideal.

[0003] Patent document CN114049357A discloses a breast ultrasound image segmentation method based on feature set correlation. The backbone network employs a U-shaped encoding / decoding structure, inputting denoised and data-enhanced images into a model for feature extraction, and calculating the loss between the output feature map and the ground truth map, as well as the feature set correlation loss. Patent document CN108665461A discloses a breast ultrasound image segmentation method based on FCN and iterative acoustic shadowing correction. A first FCN network is used to perform acoustic shadowing correction on the ultrasound image, and a second FCN network achieves the final segmentation result. These methods address the segmentation difficulties of breast ultrasound images due to noise and blurred boundaries in some images. However, the segmentation results are not ideal when artifacts or low contrast in breast ultrasound images lead to missegmentation.

[0004] In recent years, Transformer-based models have been widely used in computer vision due to their efficient self-attention mechanism, outperforming traditional Convolutional Neural Networks (CNNs). Mix Transformer (MiT) is one of many Transformer variants, with five series from MiT-B0 to MiT-B5, all with the same structure but different sizes. This invention employs a MiT-B2 network model pre-trained on the ADE20K dataset as its encoder, presenting a method for breast ultrasound mass segmentation based on Transformer and multi-level feature aggregation. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of noise in breast ultrasound images, low contrast between breast tissue and lesions, and possible artifacts and pseudostructures leading to missegmentation in breast ultrasound mass segmentation. The invention provides a breast ultrasound mass segmentation method based on Transformer and multi-level feature aggregation.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A method for segmenting breast ultrasound masses based on Transformer and multi-level feature aggregation includes the following steps:

[0008] Step 1: The MiT coding layer feature extraction module is used as the network coding part to realize feature extraction of breast ultrasound images, and outputs 4-layer feature maps F1, F2, F3 and F4;

[0009] Step 2: Input the output 4-layer feature maps into the multi-layer branch receptive field module RBF of the decoding part to obtain 4-layer feature maps R1, R2, R3 and R4 that capture broader contextual information;

[0010] Step 3: Use the permutation attention module (SAM) to capture important information from the four feature maps with broad contextual information, and output feature maps S1, S2, S3, and S4.

[0011] Step 4: Use the Aggregation Module to fuse the output feature map S3 and feature map S4 to obtain feature map A1; use the Aggregation Module to fuse feature map A1 and feature map S2 to obtain feature map A2; use the Aggregation Module to fuse feature map A2 and feature map S1 to obtain feature map A3.

[0012] The above steps are used to segment breast ultrasound images.

[0013] In step 1, the MiT coding layer feature extraction module includes a first stage, a second stage, a third stage, and a fourth stage.

[0014] like Figure 2 As shown, the feature extraction module of the coding layer is specifically as follows:

[0015] Input image → First efficient self-attention layer → First hybrid feedforward network Mix-FFN → First overlapping patch merging layer → Feature map F1 → Second efficient self-attention layer → Second hybrid feedforward network Mix-FFN → Second overlapping patch merging layer → Feature map F2 → Third efficient self-attention layer → Third hybrid feedforward network Mix-FFN → Third overlapping patch merging layer → Feature map F3 → Fourth efficient self-attention layer → Fourth hybrid feedforward network Mix-FFN → Fourth overlapping patch merging layer → Feature map F4.

[0016] In step 2, the multi-branch receptive field module (RBF) includes a first multi-branch receptive field module (RBF), a second multi-branch receptive field module (RBF), a third multi-branch receptive field module (RBF), and a fourth multi-branch receptive field module (RBF).

[0017] like Figure 3 As shown, the structure of the multi-branch receptive field module RBF model is as follows:

[0018] Feature map → 1×1 convolution operation → Feature map C1; Feature map → 1×1 convolution operation → 1×3 convolution operation → 3×1 convolution operation → 3×3 dilated convolution operation → Feature map C2; Feature map → 1×1 convolution operation → 1×5 convolution operation → 5×1 convolution operation → 3×3 dilated convolution operation → Feature map C3; Feature map → 1×1 convolution operation → 1×7 convolution operation → 7×1 convolution operation → 3×3 dilated convolution operation → Feature map C4;

[0019] Feature map C1, feature map C2, feature map C3, feature map C4 → Concatenation operation → 1×1 convolution operation → Add operation with feature map → ReLU activation function.

[0020] like Figure 4 As shown, in step 3, the permutation attention module SAM includes a first permutation attention module SAM, a second permutation attention module SAM, a third permutation attention module SAM, and a fourth permutation attention module SAM;

[0021] The steps of the SAM (Substitution Attention Module) processing are as follows:

[0022] Feature grouping → Channel attention and spatial attention → Sub-feature fusion → Feature map aggregation → Cross-group information exchange;

[0023] In this process, feature division means dividing the feature map into g groups along the channel dimension, and further dividing each group into two subgroups; channel attention means that the first subgroup outputs a channel attention sub-feature map based on the interrelationships between feature channels; spatial attention means that the second subgroup uses pixel-level spatial information to output a spatial attention sub-feature map; sub-feature fusion means performing a concatenation operation on the channel attention sub-feature map and the spatial attention sub-feature map to obtain the feature map of each group; feature map aggregation means that g groups of feature maps are obtained after sub-feature fusion processing, and then the g groups of feature maps are aggregated; cross-group information exchange means that the aggregated feature maps are subjected to a channel shuffle operation to achieve cross-group information exchange along the channel dimension.

[0024] In step 4, the aggregation module includes a first aggregation module, a second aggregation module, and a third aggregation module;

[0025] The third aggregation module aggregates feature maps S3 and S4 to obtain feature map A1; the second aggregation module aggregates feature maps A1 and S2 to obtain feature map A2; the third aggregation module aggregates feature maps A2 and S1 to obtain feature map A3.

[0026] like Figure 5 As shown, the processing steps of the Aggregation Module are as follows:

[0027] 1) Downsample the high-resolution feature map H through average pooling (Avg Pooling) to obtain feature map H1 → 3×3 convolution operation → Batch normalization (BN) operation → ReLU activation function → obtain feature map H2;

[0028] 2) Perform a 3×3 convolution operation on the high-resolution feature map H → Batch normalization operation → ReLU activation function → to obtain feature map H3;

[0029] 3) Perform bilinear interpolation upsampling on the low-resolution feature map L to obtain feature map L1 → 3×3 convolution operation → BN batch normalization operation → ReLU activation function → obtain feature map L2;

[0030] 4) Perform a 3×3 convolution operation on the low-resolution feature map L → Batch normalization operation → ReLU activation function → to obtain feature map L3;

[0031] 5) Perform add operation on feature map H2 and feature map L3 → upsampling → 3×3 convolution operation → BN batch normalization operation → ReLU activation function → to obtain feature map D1;

[0032] 6) Perform a 3×3 convolution operation on feature map H3 → Batch normalization operation → ReLU activation function → Add operation with feature map D1 → Output feature map A.

[0033] A breast ultrasound mass segmentation model based on Transformer and multi-level feature aggregation, the structure of which is as follows:

[0034] The first stage of the coding layer feature extraction module → the second stage of the coding layer feature extraction module → the third stage of the coding layer feature extraction module → the fourth stage of the coding layer feature extraction module → the multi-level feature aggregation output image of the decoding layer;

[0035] The first stage of the coding layer feature extraction module → the first multi-branch receptive field module RBF → the first permutation attention module SAM;

[0036] The second stage of the coding layer feature extraction module → the second multi-branch receptive field module RBF → the second permutation attention module SAM;

[0037] The third stage of the coding layer feature extraction module → the third multi-branch receptive field module RBF → the third permutation attention module SAM;

[0038] The fourth stage of the coding layer feature extraction module → the fourth multi-branch receptive field module RBF → the fourth permutation attention module SAM;

[0039] The third permutation attention module (SAM) and the fourth permutation attention module (SAM) are followed by the third aggregation module (AggregationModule).

[0040] The second displacement attention module (SAM) and the third aggregation module (Aggregation Module) are connected to the second aggregation module.

[0041] First displacement attention module (SAM), second aggregation module (Aggregation Module) → first aggregation module (Aggregation Module).

[0042] The coding layer feature extraction module is specifically as follows:

[0043] Input image → First efficient self-attention layer → First hybrid feedforward network Mix-FFN → First overlapping patch merging layer → Feature map F1 → Second efficient self-attention layer → Second hybrid feedforward network Mix-FFN → Second overlapping patch merging layer → Feature map F2 → Third efficient self-attention layer → Third hybrid feedforward network Mix-FFN → Third overlapping patch merging layer → Feature map F3 → Fourth efficient self-attention layer → Fourth hybrid feedforward network Mix-FFN → Fourth overlapping patch merging layer → Feature map F4.

[0044] The multi-branch receptive field module (RBF) includes a first multi-branch receptive field module (RBF), a second multi-branch receptive field module (RBF), a third multi-branch receptive field module (RBF), and a fourth multi-branch receptive field module (RBF).

[0045] The structure of the multi-branch receptive field module RBF model is as follows:

[0046] Feature map → 1×1 convolution operation → Feature map C1; Feature map → 1×1 convolution operation → 1×3 convolution operation → 3×1 convolution operation → 3×3 dilated convolution operation → Feature map C2; Feature map → 1×1 convolution operation → 1×5 convolution operation → 5×1 convolution operation → 3×3 dilated convolution operation → Feature map C3; Feature map → 1×1 convolution operation → 1×7 convolution operation → 7×1 convolution operation → 3×3 dilated convolution operation → Feature map C4;

[0047] Feature map C1, feature map C2, feature map C3, feature map C4 → Concatenation operation → 1×1 convolution operation → Add operation with feature map → ReLU activation function.

[0048] The permutation attention module (SAM) includes a first permutation attention module (SAM), a second permutation attention module (SAM), a third permutation attention module (SAM), and a fourth permutation attention module (SAM).

[0049] The steps of the SAM (Substitution Attention Module) processing are as follows:

[0050] Feature grouping → Channel attention and spatial attention → Sub-feature fusion → Feature map aggregation → Cross-group information exchange;

[0051] In this process, feature division means dividing the feature map into g groups along the channel dimension, and further dividing each group into two subgroups; channel attention means that the first subgroup outputs a channel attention sub-feature map based on the interrelationships between feature channels; spatial attention means that the second subgroup uses pixel-level spatial information to output a spatial attention sub-feature map; sub-feature fusion means performing a concatenation operation on the channel attention sub-feature map and the spatial attention sub-feature map to obtain the feature map of each group; feature map aggregation means that g groups of feature maps are obtained after sub-feature fusion processing, and then the g groups of feature maps are aggregated; cross-group information exchange means that the aggregated feature maps are subjected to a channel shuffle operation to achieve cross-group information exchange along the channel dimension.

[0052] The aggregation module includes a first aggregation module, a second aggregation module, and a third aggregation module;

[0053] The third aggregation module aggregates feature maps S3 and S4 to obtain feature map A1. The second aggregation module aggregates feature maps A1 and S2 to obtain feature map A2. The third aggregation module aggregates feature maps A2 and S1 to obtain feature map A3.

[0054] Compared with the prior art, the present invention has the following technical effects:

[0055] This invention implements a breast ultrasound mass segmentation method based on Transformer and multi-level feature aggregation. In the segmentation network, a Mix Transformer (MiT) is used as the encoding part of the model. MiT, along with the Branch Receptive Field (RBF) module, the Substitution Attention (SAM) module, and the Aggregation Module, constitute the network model. The invention was applied to segment breast ultrasound images on the publicly available BUSI dataset. Experimental results show that the Dice value reaches 85.14%, the IoU value reaches 77.54%, the Recall value reaches 86.80%, the Precision value reaches 85.05%, the Specificity value reaches 98.02%, and the Accuracy value reaches 97.1%. This surpasses currently popular breast ultrasound mass segmentation algorithms, demonstrating the superior segmentation performance of the proposed model. Attached Figure Description

[0056] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0057] Figure 1 This is a structural diagram of the breast ultrasound mass segmentation method based on Transformer and multi-level feature aggregation in this invention;

[0058] Figure 2 for Figure 1 A structural diagram of the overall structure and stages of the feature extraction module in the MiT coding layer;

[0059] Figure 3 This is a structural diagram of the branch receptive field module RBF of the present invention;

[0060] Figure 4 This is a structural diagram of the displacement attention module (SAM) of the present invention;

[0061] Figure 5 This is a structural diagram of the Aggregation Module of the present invention;

[0062] Figure 6 A comparison of breast mass segmentation results using the BUS dataset. Detailed Implementation

[0063] A method for segmenting breast ultrasound masses based on Transformer and multi-level feature aggregation includes the following steps:

[0064] Step 1: The MiT coding layer feature extraction module is used as the network coding part to extract features from the breast ultrasound image and output four feature maps F1, F2, F3, and F4.

[0065] Step 2: Input the output 4-layer feature maps into the multi-layer branch receptive field module RBF of the decoding part to obtain 4-layer feature maps R1, R2, R3, and R4 that capture broader contextual information;

[0066] Step 3: Use the permutation attention module (SAM) to capture important information from the four feature maps with broad contextual information, and output feature maps S1, S2, S3, and S4.

[0067] Step 4: The aggregation module is used to perform a stepwise fusion operation on the output feature maps S1, S2, S3 and S4 of the permutation attention module SAM, and output feature map A3.

[0068] The above steps are used to segment breast ultrasound images.

[0069] In step 1, the MiT coding layer feature extraction module includes a first stage, a second stage, a third stage, and a fourth stage.

[0070] The coding layer feature extraction module is specifically as follows:

[0071] Feature map → Efficient Self-Attention layer → Mix-FFN (Hybrid Feedforward Network) → Overlap Patch Merging layer;

[0072] In step 2, the multi-branch receptive field module (RBF) includes a first multi-branch receptive field module (RBF), a second multi-branch receptive field module (RBF), a third multi-branch receptive field module (RBF), and a fourth multi-branch receptive field module (RBF).

[0073] The steps for RBF processing in the multi-branch receptive field module are as follows:

[0074] 1) Parallel branch 1 performs a 1×1 convolution operation on the feature map;

[0075] 2) Parallel branch 2 performs 1×1 convolution operation → 1×3 convolution operation → 3×1 convolution operation → 3×3 dilated convolution operation on the feature map with a dilation coefficient of 3;

[0076] 3) Parallel branch 3 performs 1×1 convolution operation on the feature map → 1×5 convolution operation → 5×1 convolution operation → 3×3 dilated convolution operation with a dilation coefficient of 5;

[0077] 4) Parallel branch 4 performs 1×1 convolution operation on the feature map → 1×7 convolution operation → 7×1 convolution operation → 3×3 dilated convolution operation with a dilation coefficient of 7.

[0078] 5) Perform a concatenation operation on the outputs of 1), 2), 3), and 4) → 1×1 convolution operation

[0079] 6) Add the feature map to the output of 5) → ReLU activation function

[0080] In step 3, the permutation attention module SAM includes a first permutation attention module SAM, a second permutation attention module SAM, a third permutation attention module SAM, and a fourth permutation attention module SAM;

[0081] The steps of the SAM (Substitution Attention Module) processing are as follows:

[0082] 6) Feature grouping: Divide the feature map into g groups along the channel dimension, and further divide each group into two subgroups;

[0083] 7) Channel attention: The first group outputs a channel attention sub-feature map based on the interrelationships between feature channels;

[0084] 8) Spatial attention: The second group utilizes pixel-level spatial information to output a spatial attention sub-feature map;

[0085] 9) Sub-feature fusion: Perform a concatenation fusion operation on the channel attention sub-feature map in 2) and the spatial attention sub-feature map in 3) to obtain the feature map of each group;

[0086] 10) Feature map aggregation: After processing in step 4), g sets of feature maps are obtained, and then the g sets of feature maps are aggregated.

[0087] 11) Cross-group information exchange: Channel shuffle operation is used on the aggregated feature map to realize cross-group information exchange along the channel dimension;

[0088] In step 4, the aggregation module includes a first aggregation module, a second aggregation module, and a third aggregation module;

[0089] The third aggregation module aggregates feature maps S3 and S4 to obtain feature map A1; the second aggregation module aggregates feature maps A1 and S2 to obtain feature map A2; the third aggregation module aggregates feature maps A2 and S1 to obtain feature map A3.

[0090] The processing steps of the Aggregation Module are as follows:

[0091] 1) Downsample the high-resolution feature map H through average pooling (Avg Pooling) to obtain feature map H1 → 3×3 convolution operation → Batch normalization (BN) operation → ReLU activation function → obtain feature map H2;

[0092] 2) Perform a 3×3 convolution operation on the high-resolution feature map H → Batch normalization operation → ReLU activation function → to obtain feature map H3;

[0093] 3) Perform bilinear interpolation upsampling on the low-resolution feature map L to obtain feature map L1 → 3×3 convolution operation → BN batch normalization operation → ReLU activation function → obtain feature map L2;

[0094] 4) Perform a 3×3 convolution operation on the low-resolution feature map L → Batch normalization operation → ReLU activation function → to obtain feature map L3;

[0095] 5) Perform add operation on feature map H2 and feature map L3 → upsampling → 3×3 convolution operation → BN batch normalization operation → ReLU activation function → to obtain feature map D1;

[0096] 6) Perform a 3×3 convolution operation on feature map H3 → Batch normalization operation → ReLU activation function → Add operation with feature map D1 → Output feature map A;

[0097] This invention also includes a breast ultrasound mass segmentation model based on Transformer and multi-level feature aggregation, the structure of which is as follows:

[0098] The first stage of the coding layer feature extraction module 1 → the second stage of the coding layer feature extraction module 2 → the third stage of the coding layer feature extraction module 3 → the fourth stage of the coding layer feature extraction module 4 → the multi-level feature aggregation output image of the decoding layer 5;

[0099] The first stage of the coding layer feature extraction module is 1 → the first multi-layer branch receptive field module RBF6 → the first permutation attention module SAM10;

[0100] The second stage of the coding layer feature extraction module 2 → the second multi-layer branch receptive field module RBF7 → the second permutation attention module SAM11;

[0101] The third stage of the coding layer feature extraction module is 3 → the third multi-layer branch receptive field module RBF8 → the third permutation attention module SAM12;

[0102] The fourth stage of the coding layer feature extraction module is 4 → the fourth multi-layer branch receptive field module RBF9 → the fourth permutation attention module SAM13;

[0103] The third permutation attention module SAM12, the fourth permutation attention module SAM13 → the third aggregation module Aggregation Module 14;

[0104] Second displacement attention module SAM11, third aggregation module Aggregation Module14 → second aggregation module Aggregation Module15;

[0105] First displacement attention module SAM10, second aggregation module Aggregation Module15 → first aggregation module Aggregation Module16.

[0106] The coding layer feature extraction module is specifically as follows:

[0107] Input image → First efficient self-attention layer → First hybrid feedforward network Mix-FFN → First overlapping patch merging layer → Feature map F1 → Second efficient self-attention layer → Second hybrid feedforward network Mix-FFN → Second overlapping patch merging layer → Feature map F2 → Third efficient self-attention layer → Third hybrid feedforward network Mix-FFN → Third overlapping patch merging layer → Feature map F3 → Fourth efficient self-attention layer → Fourth hybrid feedforward network Mix-FFN → Fourth overlapping patch merging layer → Feature map F4.

[0108] The feature extraction module of the coding layer is used to extract the features of the mass in the breast ultrasound image layer by layer. The extracted features include both high-resolution fine features and low-resolution coarse features. The rich feature extraction is beneficial to the subsequent segmentation of the breast mass.

[0109] The multi-branch receptive field module (RBF) includes a first multi-branch receptive field module (RBF), a second multi-branch receptive field module (RBF), a third multi-branch receptive field module (RBF), and a fourth multi-branch receptive field module (RBF).

[0110] The structure of the multi-branch receptive field module RBF model is as follows:

[0111] Feature map → 1×1 convolution operation → Feature map C1; Feature map → 1×1 convolution operation → 1×3 convolution operation → 3×1 convolution operation → 3×3 dilated convolution operation with a dilation coefficient of 3 → Feature map C2; Feature map → 1×1 convolution operation → 1×5 convolution operation → 5×1 convolution operation → 3×3 dilated convolution operation with a dilation coefficient of 5 → Feature map C3; Feature map → 1×1 convolution operation → 1×7 convolution operation → 7×1 convolution operation → 3×3 dilated convolution operation with a dilation coefficient of 7 → Feature map C4;

[0112] Feature map C1, feature map C2, feature map C3, feature map C4 → Concatenation operation → 1×1 convolution operation → Add operation with feature map → ReLU activation function.

[0113] Among them, the multi-branch receptive field module (RBF) is used to capture feature information of target regions under different receptive fields. The acquired multi-scale features achieve a full combination of contextual semantic information and spatial details, solve the problem of low contrast between breast mass regions and surrounding tissues, and reduce the possibility of missegmentation.

[0114] The permutation attention module (SAM) includes a first permutation attention module (SAM), a second permutation attention module (SAM), a third permutation attention module (SAM), and a fourth permutation attention module (SAM).

[0115] The steps of the SAM (Substitution Attention Module) processing are as follows:

[0116] Feature grouping 21 → Channel attention and spatial attention 22 → Sub-feature fusion 23 → Feature map aggregation 24 → Cross-group information exchange 25;

[0117] In this process, feature division means dividing the feature map into g groups along the channel dimension, and further dividing each group into two subgroups; channel attention means that the first subgroup outputs a channel attention sub-feature map based on the interrelationships between feature channels; spatial attention means that the second subgroup uses pixel-level spatial information to output a spatial attention sub-feature map; sub-feature fusion means performing a concatenation operation on the channel attention sub-feature map and the spatial attention sub-feature map to obtain the feature map of each group; feature map aggregation means that g groups of feature maps are obtained after sub-feature fusion processing, and then the g groups of feature maps are aggregated; cross-group information exchange means that the aggregated feature maps are subjected to a channel shuffle operation to achieve cross-group information exchange along the channel dimension.

[0118] Among them, the displacement attention module (SAM) extracts feature information of breast lumps along the channel and spatial dimensions, effectively capturing information such as the texture and shape of the lumps, while suppressing the influence of noise and irrelevant regions, enhancing the model's detection and localization of the lump region, and improving the accuracy and robustness of the segmentation model.

[0119] The aggregation module includes a first aggregation module, a second aggregation module, and a third aggregation module;

[0120] The third aggregation module aggregates feature maps S3 and S4 to obtain feature map A1. The second aggregation module aggregates feature maps A1 and S2 to obtain feature map A2. The third aggregation module aggregates feature maps A2 and S1 to obtain feature map A3.

[0121] The aggregation module integrates deep and shallow features processed by the multi-branch receptive field module (RBF) and the substitution attention module (SAM), weakening the semantic gap between features, further refining the edge details of the breast mass, ensuring the continuity of the mass region segmentation, and thus improving the segmentation accuracy.

[0122] To illustrate the segmentation performance of the model constructed in this invention, comparative experiments were conducted with U-Net, U-Net++, AttentionUNet, TransUnet, and TransFuse network models. The Dice coefficient, Intersection over Union (IoU), Recall, Precision, Specificity, and Accuracy were used as evaluation metrics. These six metrics are commonly used to evaluate the segmentation performance of models in medical image segmentation tasks. The model of this invention was applied to the public datasets BUSI, Dataset B, and BUS to segment breast ultrasound images of masses. The BUSI dataset is from Baheya Women's Hospital in Egypt; this experiment used only 583 images of benign masses and 64 images of malignant masses. Dataset B is from the Cancer Center of Sun Yat-sen University, containing 160 images of both benign and malignant masses. The BUS dataset is from the UDIAT Diagnostic Center, containing 110 images of benign masses and 53 images of malignant masses. The evaluation metrics of the model of this invention and other breast ultrasound image mass segmentation models on the BUSI dataset are shown in Table 1. The evaluation metrics for Dataset B are shown in Table 2.

[0123] Table 1 shows the segmentation results on the BUSI dataset.

[0124]

[0125] Table 2 shows the results of the partitioning on Dataset B.

[0126]

[0127] Experimental results show that the model of this invention outperforms the aforementioned network models for breast ultrasound image mass segmentation on both the BUSI and Dataset B public datasets, achieving the best results in Dice coefficient, Intersection over Union (IoU), Recall, Precision, Specificity, and Accuracy. In the BUSI dataset, compared to the TransFuse network, the model of this invention improves Dice by 3.21%, IoU by 3.19%, Recall by 3.19%, Precision by 2.59%, Specificity by 1.27%, and Accuracy by 0.52%. In Dataset B, compared to the TransFuse network, the model of this invention improves Dice by 1.83%, IoU by 3.23%, Recall by 1.37%, Precision by 5.16%, Specificity by 4.4%, and Accuracy by 1.78%. Therefore, the model of this invention has certain advantages over other medical image segmentation methods for breast ultrasound image mass segmentation.

[0128] Figure 6 The results show a comparison of breast mass segmentation performance on the BUS dataset. It is evident that the model in this invention effectively addresses the problem of low contrast between the breast mass and surrounding tissue. This is because the use of a multi-layered branch receptive field module allows for the full learning of the breast mass's feature information, such as edges, texture, and shape. Figure 6 As can be seen from the diagram, the model in this invention utilizes the Permutation Attention Module (SAM) to focus more on the target region, suppressing the influence of noise and irrelevant information, thus achieving accurate localization of the breast lump region. In contrast, the segmentation results of Unet, Attention Unet, and Unet++ models suffer from missegmentation and omissions, resulting in poor segmentation performance. Furthermore, the model in this invention achieves continuity in breast lump segmentation by using the Aggregation Module to aggregate the enhanced feature maps obtained from the Permutation Attention Module (SAM) layer by layer, fusing deep and shallow features to avoid discontinuous segmentation and improve segmentation accuracy.

[0129] In summary, this invention proposes a breast ultrasound mass segmentation model based on Transformer and multi-level feature aggregation. Unlike existing models based on U-shaped network designs, the encoding part employs a Mix Transformer to fully capture both high-resolution coarse features and low-resolution fine features of the breast mass, simultaneously obtaining multi-scale feature maps. In the decoding part, a multi-branch receptive field module (RFB) is used to acquire feature information of the breast mass under different receptive fields, fully learning the features of the mass in the image, such as edges, texture, and shape. A permutation attention module (SAM) is used to focus more on the target region, enhancing the key features of the mass and improving the detection performance of the mass region. The aggregation module aggregates the enhanced feature maps obtained from the permutation attention module SAM layer by layer, achieving the fusion of deep and shallow features, constructing a connection between local and global features, and improving the accuracy of breast mass segmentation.

Claims

1. A method for segmenting breast ultrasound masses based on Transformer and multi-level feature aggregation, comprising the following steps: Step 1: The MiT coding layer feature extraction module is used as the network coding part to realize feature extraction of breast ultrasound images, and outputs 4-layer feature maps F1, F2, F3 and F4; Step 2: Input the output 4-layer feature maps into the multi-layer branch receptive field module RBF of the decoding part to obtain 4-layer feature maps R1, R2, R3 and R4 that capture broader contextual information; Step 3: Use the permutation attention module (SAM) to capture important information from the four feature maps with broad contextual information, and output feature maps S1, S2, S3, and S4. Step 4: Use the Aggregation Module to fuse the output feature map S3 and feature map S4 to obtain feature map A1; use the Aggregation Module to fuse feature map A1 and feature map S2 to obtain feature map A2; use the Aggregation Module to fuse feature map A2 and feature map S1 to obtain feature map A3. The above steps are used to segment breast ultrasound images.

2. The method according to claim 1, characterized in that, In step 1, the MiT coding layer feature extraction module includes a first stage (1), a second stage (2), a third stage (3), and a fourth stage (4) of the coding layer feature extraction module. The coding layer feature extraction module is specifically as follows: Input image → First efficient self-attention layer → First hybrid feedforward network Mix-FFN → First overlapping patch merging layer → Feature map F1 → Second efficient self-attention layer → Second hybrid feedforward network Mix-FFN → Second overlapping patch merging layer → Feature map F2 → Third efficient self-attention layer → Third hybrid feedforward network Mix-FFN → Third overlapping patch merging layer → Feature map F3 → Fourth efficient self-attention layer → Fourth hybrid feedforward network Mix-FFN → Fourth overlapping patch merging layer → Feature map F4.

3. The method according to claim 1, characterized in that, In step 2, the multi-branch receptive field module (RBF) includes a first multi-branch receptive field module (RBF), a second multi-branch receptive field module (RBF), a third multi-branch receptive field module (RBF), and a fourth multi-branch receptive field module (RBF). The structure of the multi-branch receptive field module RBF model is as follows: Feature map → 1×1 convolution operation → Feature map C1 (17); Feature map → 1×1 convolution operation → 1×3 convolution operation → 3×1 convolution operation → 3×3 dilated convolution operation with dilation coefficient of 3 → Feature map C2 (18); Feature map → 1×1 convolution operation → 1×5 convolution operation → 5×1 convolution operation → 3×3 dilated convolution operation with dilation coefficient of 5 → Feature map C3 (19); Feature map → 1×1 convolution operation → 1×7 convolution operation → 7×1 convolution operation → 3×3 dilated convolution operation with dilation coefficient of 7 → Feature map C4 (20); Feature map C1, feature map C2, feature map C3, feature map C4 → Concatenation operation → 1×1 convolution operation → Add operation with feature map → ReLU activation function.

4. The method according to claim 1, characterized in that, In step 3, the permutation attention module SAM includes a first permutation attention module SAM, a second permutation attention module SAM, a third permutation attention module SAM, and a fourth permutation attention module SAM; The steps of the displacement attention module (SAM) processing are as follows: Feature grouping (21) → Channel attention and spatial attention (22) → Sub-feature fusion (23) → Feature map aggregation (24) → Cross-group information exchange (25) In this process, feature division means dividing the feature map into g groups along the channel dimension, and further dividing each group into two subgroups; channel attention means that the first subgroup outputs a channel attention sub-feature map based on the interrelationships between feature channels; spatial attention means that the second subgroup uses pixel-level spatial information to output a spatial attention sub-feature map; sub-feature fusion means performing a concatenation operation on the channel attention sub-feature map and the spatial attention sub-feature map to obtain the feature map of each group; feature map aggregation means that g groups of feature maps are obtained after sub-feature fusion processing, and then the g groups of feature maps are aggregated; cross-group information exchange means that the aggregated feature maps are subjected to a channel shuffle operation to achieve cross-group information exchange along the channel dimension.

5. The method according to claim 1, characterized in that, In step 4, the aggregation module includes a first aggregation module, a second aggregation module, and a third aggregation module; The third aggregation module aggregates feature maps S3 and S4 to obtain feature map A1; the second aggregation module aggregates feature maps A1 and S2 to obtain feature map A2; the third aggregation module aggregates feature maps A2 and S1 to obtain feature map A3. The Aggregation Module processes data as follows: 1) Downsample the high-resolution feature map H through average pooling (Avg Pooling) to obtain feature map H1 → 3×3 convolution operation → Batch normalization (BN) operation → ReLU activation function → obtain feature map H2; 2) Perform a 3×3 convolution operation on the high-resolution feature map H → Batch normalization operation → ReLU activation function → to obtain feature map H3; 3) Perform bilinear interpolation upsampling on the low-resolution feature map L to obtain feature map L1 → 3×3 convolution operation → BN batch normalization operation → ReLU activation function → obtain feature map L2; 4) Perform a 3×3 convolution operation on the low-resolution feature map L → Batch normalization operation → ReLU activation function → to obtain feature map L3; 5) Perform add operation on feature map H2 and feature map L3 → upsampling → 3×3 convolution operation → BN batch normalization operation → ReLU activation function → to obtain feature map C1; 6) Perform a 3×3 convolution operation on feature map H3 → Batch normalization operation → ReLU activation function → Add operation with feature map C1 → Output feature map A.

Citation Information

Patent Citations

  • Breast ultrasound image segmentation method based on FCN and iterative acoustic shadow correction

    CN108665461A

  • Breast ultrasonic segmentation method based on feature set correlation degree

    CN114049357A

  • U-shaped self-adaptive EST unmanned aerial vehicle remote sensing image segmentation method and system

    CN115035131A

  • Refined image semantic segmentation method based on progressive neighbor aggregation

    CN116246065A