Weak supervision breast ultrasound image segmentation method based on strategy fusion and SAM optimization segmentation

Abstract

The invention discloses a weak supervision breast ultrasound image segmentation method based on strategy fusion and SAM optimization segmentation, and belongs to the field of computer vision. The method is generally divided into two stages of preliminary segmentation and optimized segmentation. The first stage is preliminary segmentation, and two parallel branches are complementarily fused to form a preliminary segmentation result. Wherein the boundary extraction branch extracts boundary information of a target area structure through a morphological segmentation method, the semantic information extraction branch realizes image semantic feature extraction through a training classification network, a heat map is generated by adopting a method based on LayerCAM to highlight a target segmentation area, a segmentation mask is obtained from the heat map by using a threshold segmentation method, and the target segmentation area is subjected to image segmentation. And then fusing the masks of the two branches by adopting a fusion strategy to form a preliminary segmentation result. In the optimal segmentation stage, the SAM model serves as a segmentation enhancement tool to refine the pseudo tag after preliminary segmentation, and a post-processing step is used for further enhancement, so that the segmentation accuracy is improved.

Description

Technical Field

[0001] The invention designs a weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimization segmentation, belonging to the field of computer vision. Background Art

[0002] Breast ultrasound image segmentation is a key computer vision task that aims to automatically identify and locate mass areas in the breast, such as tumors or abnormal tissues. Ultrasound imaging, with its non-invasive and radiation-free characteristics, plays an irreplaceable role in breast image segmentation tasks. However, ultrasound images usually have low contrast, accompanied by high noise and blurred edges, which further increases the complexity of automatic segmentation.

[0003] With the improvement of computer hardware performance and the rapid development of deep learning technology, especially convolutional neural networks (CNN), medical image segmentation has entered a new stage of development. Deep learning models such as U-Net and FCN have become the mainstream methods in the field of medical image segmentation. These networks are able to learn complex feature representations and achieve more accurate image segmentation through end-to-end training. At present, most breast ultrasound image segmentation models rely on supervised learning methods. It mainly uses training data with pixel-level labels to train models for prediction or segmentation. For example, Qin Chuanbo proposed a segmentation model based on U-Net and attention-residual, and verified it on a self-built breast cancer image dataset of 160 cases, with a Dice value of 77.92. However, this method requires a large amount of labeled training data, and the annotation of medical images usually requires manual annotation by professionals, which is costly and time-consuming, so it is a challenge to obtain sufficient quantity and quality of labeled data.

[0004] Compared with supervised segmentation methods, weakly supervised learning methods can help solve the problem of scarce or expensive labeled data. It uses less, partially labeled or inaccurately labeled data than fully labeled data to guide model learning during training. For example, the paper "Weakly Supervised Deep Learning for Breast Cancer Segmentation with Coarse Annotations" by Hao Zheng et al. proposes a weakly supervised segmentation framework that uses rough annotations generated by boundary graffiti to train deep convolutional neural networks.

[0005] Recently, the Segment Anything Model (SAM) has attracted much attention in the computer vision community. SAM is a cue-based segmentation model that can zero-shot generalize to unfamiliar objects and images without additional training. There is growing interest in extending SAM to the field of medical imaging.

[0006] Although the above methods can complete the breast ultrasound image segmentation task, the existing segmentation methods still have the following challenges: (1) Supervised learning requires a large amount of labeled training data, and the cost of creating annotations for large breast ultrasound image datasets is high. (2) Existing weakly supervised learning methods usually require the design of complex loss functions or training strategies to effectively utilize weak supervision signals, which leads to difficult convergence of the training process and long training time. (3) SAM is mainly aimed at natural images. Directly using SAM for medical image segmentation cannot obtain accurate segmentation results. Based on the above limitations, a weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimized segmentation is proposed. Summary of the invention

[0007] A weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimized segmentation

[0008] In view of the shortcomings of the prior art, the present invention proposes a weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimization segmentation. Figure 1 As shown. The method is generally divided into two stages: preliminary segmentation and optimized segmentation. Stage one is preliminary segmentation, which uses two parallel branches to complement each other to form a preliminary segmentation result. The boundary extraction branch extracts the structural boundary information of the target area through the morphological segmentation method, and the semantic information extraction branch realizes the image semantic feature extraction by training the classification network. The LayerCAM-based method is used to generate a heat map to highlight the target segmentation area. The threshold segmentation method is used to obtain the segmentation mask from the heat map, and then the fusion strategy is used to fuse the masks of the two branches to form a preliminary segmentation result. In the optimized segmentation stage, the SAM model is used as a segmentation enhancement tool to refine the pseudo-labels after the preliminary segmentation, and the post-processing steps are used for further enhancement to improve the segmentation accuracy.

[0009] In order to achieve the above invention object, the present invention adopts the following technical scheme:

[0010] A weakly supervised medical image segmentation method based on SAM optimized segmentation includes: a data preprocessing module, a boundary extraction module, a semantic information extraction module, a mask fusion module, and a SAM optimized segmentation module.

[0011] Data preprocessing module: This module consists of two parts, image resizing and image set division. First, all images are resized to 256×256 for subsequent segmentation. Then, the image set is divided into a training set and a test set. The training set and the test set contain two types of data, images with masses and normal images.

[0012] Boundary extraction module: This module uses the K-means clustering algorithm to obtain a binary image, retaining the main features and boundary information. It determines the height and width of the suspected target area, and then uses morphological knowledge to filter the non-target area.

[0013] Semantic Information Extraction Module: This module uses image-level labels to extract semantic information. The image training set is put into the classification model DenseNet for classification training, and an optimized class activation mapping (CAM) method, namely LayerCAM, is used to generate image heat maps and binary images.

[0014] Mask fusion module: This module applies the mask fusion strategy to complementarily fuse the results of the boundary information extraction module and the results of the semantic information extraction module, and uses the advantages of both to improve the initial segmentation accuracy.

[0015] SAM optimized segmentation module: This stage extracts frame hint information from the preliminary segmentation results. For each preliminary segmented target area, the minimum bounding rectangle of the area is generated, and the upper left corner coordinates and lower right corner coordinates of the bounding rectangle are extracted as frame hint information. Subsequently, the original breast ultrasound image and frame hint information are sent to SAM to enhance the preliminary segmentation results of the target area. Finally, morphological operations are used to fill the hole areas formed in the SAM segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is the overall architecture diagram of the weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimized segmentation.

[0017] Figure 2 This is the result of visualizing the prediction results on the BUSI dataset.

[0018] The specific implementation and exemplary embodiments of each step of the present invention will be described in detail below

[0019] Step 1: Data Preprocessing

[0020] Data preprocessing mainly involves image resizing and image set division.

[0021] Step 1.1: Resize the screened breast ultrasound images to 256×256.

[0022] Step 1.2: Randomly divide the image set into a training set and a test set with a ratio of 8:2. Both the training set and the test set contain two types of data, namely breast ultrasound images with masses and normal images.

[0023] Step 2: Boundary extraction

[0024] Step 2.1: Since the overall brightness of breast ultrasound images is dim and the grayscale values ​​are roughly distributed in a lower range, ACE (automatic color enhancement) operation is first performed on them. The final pixel value is corrected by calculating the brightness and darkness of the target pixel and the surrounding pixels in the image and their relationship to achieve image contrast adjustment. Let I represent the given color channel of the RGB image, x is the target pixel coordinate in the image, y is the neighborhood pixel of the current pixel x, I(x) represents the pixel value with coordinate x, and I(y) represents the pixel value with coordinate y, and its domain Ω and intensity value are normalized to the range [0,1]. The chromatic aberration correction operations performed independently on the three channels are as follows, and R(x) is the enhanced pixel value:

[0025]

[0026] Where Ω\x represents {y∈Ω∶y≠x}, ‖xy‖ represents the Euclidean distance, s α (*) represents the slope function, which is used to limit the enhancement effect of the difference I(x)-I(y) and is defined as:

[0027] s α (t)=min{max{αt,-1},1} (2)

[0028] Among them, α≥1 is a parameter used to control the slope, which affects the enhancement effect. t is the pixel value difference between the target pixel x and the neighboring pixel y, which is used to measure the difference in brightness or color value. This step is adapted to local image contrast, s α (*) It can amplify small differences and enrich large differences, expand or compress the dynamic range according to the local content, and then normalize R to [0,1].

[0029] Step 2.2: Then, the K-means clustering algorithm is used to divide the target area and the non-target area to obtain a binary segmentation mask. The clustering algorithm shown in the objective function D is used to achieve optimal clustering, where k represents the number of clusters, n represents the number of points, and (x j -c i ) 2 For data point x j and its cluster centroid c i quantified results.

[0030]

[0031] Step 2.3: In order to filter the suspected target areas in the breast parenchyma layer, this step uses morphological knowledge to filter out the incorrectly extracted tissue aspect ratio. The height and width are determined by calculating the minimum enclosing rectangle of each suspected target area, and then it is filtered using morphological knowledge, thereby obtaining a more accurate segmentation result. d) represents the aspect ratio of the dth suspected target area in the binary image.

[0032]

[0033] Step 3: Semantic Information Extraction

[0034] In this stage, a deep learning network is used to extract semantic information from the image and generate a segmentation mask containing semantic feature information.

[0035] Step 3.1: In order to extract a set of powerful and representative features, this stage uses the DenseNet-121 network to classify medical images. DenseNet-121 is a specific variant of the DenseNet architecture with 121 layers. In this network, multiple convolutional layers within a dense block fully utilize the features of the previous layer and gradually build richer semantic information, which promotes feature reuse and information transfer. Let U = {X i ,Y i} represents a medical image dataset, where X i is the i-th image, Y i is its corresponding image-level label, indicating that X i Whether there is a mass area in the . Then the data with a mass is expressed as B = ((X i ,Y i )∈U|Y i =1), the rest of the data is expressed as N=((X i ,Y i )∈U|Y i =0). On this basis, by minimizing the loss L cls To train a deep classification network f:

[0036]

[0037] Where m represents the number of samples, f represents the classification network, and s(*) is the Sigmoid function.

[0038] The classification model we used, DenseNet121, was pre-trained on ImageNet, using stochastic gradient descent (SGD) as the optimizer with a weight decay of 0.0004 and a momentum of 0.9. The learning rate started at 1e-4 and the model was trained for 100 epochs with a batch size of 16.

[0039] Step 3.2: Integrate LayerCAM after the last convolutional layer of DenseNet121 to generate a heat map of the breast ultrasound image. For example, given a breast ultrasound image I u, input into the image classifier to obtain the output feature map of the last convolutional layer and the predicted classification score of the target category.

[0040] y c =f(I u ,θ N ) (6)

[0041] where y c is the predicted classification score, f is the classification network, θ N is the classification network parameter, c is the category. Assume that M represents the feature map of the last convolutional layer, M e represents the e-th feature map in the feature map M, represents the activation value of the e-th feature map at position (i, j). Next, the predicted score of the target category is calculated relative to M e The gradient at a specific spatial position (i,j) in:

[0042]

[0043] A positive gradient corresponding to a position in the feature map indicates that increasing the strength of that position will have a positive impact on the prediction score of the target class. Therefore, for positions with positive gradients, their gradients are used as weights in LayerCAM. For positions with negative gradients, the weights are assigned to zero.

[0044]

[0045] Based on the above operations, LayerCAM not only considers the importance of different channels, but also the importance of different spatial positions. The individual weights of each position can reflect the importance of different positions in the feature map related to the target class. In order to obtain the class activation map of a certain layer, LayerCAM first multiplies the activation value of each position in the feature map by a weight. The weighted activation value of each position in the feature map is

[0046]

[0047] in, Indicates the use of weights Weighted, get the adjusted weighted activation value. Finally, linearly combine the above results along the channel dimension to get the class activation map:

[0048]

[0049] in Indicates M eThe feature map obtained after weighting. This type of activation map can capture reliable preliminary segmentation and positioning information. LayerCAM effectively uses deep learning to associate key image features with target areas, thereby obtaining valuable segmentation and positioning information.

[0050] Step 3.3: Use the global threshold segmentation algorithm to convert the continuous value class activation map into a binary image. Since the LayerCAM method activates the key area in red, for an activated breast ultrasound thermal image A i , extract its red channel data R(A i ), and calculate its histogram to obtain the frequency distribution of each pixel value:

[0051]

[0052] Where hist is the histogram calculation operation. Next, extract all pixel values ​​within interval P, where P is the red pixel value interval of the activation heat map, specifically P = [P min ,P max ], where P min =180, P max =255:

[0053]

[0054] Where S i is a set of pixel values ​​p within the pixel value interval P. Then the mean value of red pixels in a single activation map is calculated:

[0055]

[0056] And finally the average pixel value of all images is obtained as the binarization threshold μ:

[0057]

[0058] Where N is the number of activated images in the corresponding data set, the pixel value at position (x, y) on the pseudo label is:

[0059]

[0060] Where C is the category, CAM C (x,y) represents the pixel value at position (x,y) on the class activation map. Through this thresholding process, a pseudo label that preliminarily identifies the location and size of the target area is obtained without any additional annotation data.

[0061] Step 4: Segmentation mask fusion

[0062] The K-means algorithm has a remarkable ability to extract the contour of the target, however, the K-means algorithm is sensitive to noise, and noise points may be incorrectly assigned to clusters. Therefore, it is not feasible to use only the traditional segmentation method based on K-means. In contrast, the deep classification network is able to locate the most prominent area of ​​the target through CAM, however, it is affected by inaccurate activation. Therefore, this step integrates the advantages of both methods and constructs a more accurate target segmentation area, thereby significantly improving the accuracy of segmentation.

[0063] Assumptions represents the set of all suspicious regions extracted from a breast ultrasound image by boundary extraction, and represents the set of all suspicious regions extracted by the semantic feature extraction module. We will M Each target contour in is extracted and then compared with L C The target area in the image is integrated to find L M Middle and L C The area with the largest intersection is considered as the fusion result. M All suspected target areas and L C If there is no intersection between them, then the set L C is regarded as our fusion result. The above operation can be expressed as:

[0064]

[0065] Step 5: SAM Optimized Segmentation

[0066] Step 5.1: This step extracts the box hint information from the fused pseudo-label. For a breast ultrasound image, traverse all connected regions marked as 1 on the pseudo-label map and find the largest connected region on the image. Calculate its minimum enclosing rectangle. Where N represents the number of pixels in the connected area, (x i ,y i ) represents the coordinates of the i-th pixel in the connected area. Calculate the coordinates of the upper left corner of its minimum bounding rectangle (x min ,y min ) and the lower right corner coordinate (x max ,y max ):

[0067] x min =min{x i};y min =min{y i} (17)

[0068] x max =max{x i};y max=max{y i} (18)

[0069] Then calculate the width w and height h of the minimum enclosing rectangle:

[0070] w=x max -x min (19)

[0071] h=y max -y min (20)

[0072] Step 5.2: Use the original breast ultrasound image as the input of SAM and send it to the image encoder. Send the box hint information obtained in the previous step as the seed region signal to the hint encoder of SAM. The hint encoder encodes the bounding box information into the embedding vector. Finally, the lightweight mask decoder predicts the embedding from the breast ultrasound image and the hint encoder to obtain the optimized segmented mask.

[0073] Step 5.3: Post-processing of segmentation results

[0074] This step uses morphological post-processing operations to fill the hole areas in the SAM output results. The specific method is to use a 3×3 cross-shaped structure element to perform an iterative dilation operation, update the segmentation mask obtained in the previous stage through distance-based transformation iterations, extract and fill the segmentation mask, and obtain the final segmentation result.

[0075] In step 5, the Dice similarity coefficient (Dice), 95% Hausdorff distance (HD95), and intersection over union (IoU) were used to evaluate the performance of the method. The mean and 95% confidence interval were calculated for each indicator. The confidence interval was determined using bootstrap analysis, including 5000 resampling iterations to ensure statistical robustness and accuracy of the results.

[0076] In this study, different model algorithms are compared, and the experimental results are shown in Table 1:

[0077] Table 1 Experimental results of different model algorithms on the BUSI dataset

[0078]

[0079] In the table above, the performance differences between the proposed method and the other three methods on the BUSI dataset are quantitatively compared in terms of three evaluation indicators, where SL represents the fully supervised scenario and WSL represents the weakly supervised scenario. The proposed method achieves comparable results to the fully supervised segmentation model using only class-level labels, and the HD95 indicator is better than the Deeplabv3+ segmentation model in the fully supervised scenario. In the weakly supervised scenario, all indicators are far better than the AffinityNet segmentation model.

[0080] In order to verify the effectiveness of different modules, we conduct ablation experiments on the BUSI dataset for verification, where EdgeSeg represents the boundary extraction module and LCAM represents the semantic information extraction module. The results are shown in Table 2:

[0081] Table 2 Ablation experiment results on the BUSI dataset

[0082]

[0083] It can be seen from Table 2 that each module improves the segmentation performance of the model to a certain extent.

[0084] In order to intuitively compare the segmentation effects of different model algorithms, we visualize the prediction results of UNet, Deeplabv3+, AffinityNet and the method we studied on the BUSI dataset, as shown in the figure below: Figure 2 As shown in the figure, it can be seen intuitively that AffinityNet has the lowest segmentation accuracy, almost no effective target area is segmented, and a large area of ​​erroneous prediction area occurs; UNet performs relatively well in boundary segmentation, but when predicting some breast ultrasound images, it does not predict the complete target area; Deeplabv3+'s prediction is over-segmented, and the boundary area cannot be accurately segmented; in contrast, the method we studied shows a more accurate segmentation result, and can pay more attention to the boundary features of the target area. This ability to capture boundaries further highlights the effectiveness of our method.

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] 1. The weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimized segmentation adopted by the present invention only relies on image class-level labels and effectively segments the target area without providing pixel-level labels.

[0087] Second, the weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimized segmentation adopted by the present invention integrates the boundary information, semantic features and precise segmentation capability of breast ultrasound images. Through these different methods, the framework can learn more comprehensive image features and make full use of the comprehensive advantages of each method.

[0088] 3. The weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimized segmentation adopted in the present invention provides some valuable ideas and guidance for the application of SAM in the field of medical images, which helps to improve the performance of SAM in medical image segmentation tasks.

Claims

1. A weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimized segmentation, characterized in that It includes the following five modules: data preprocessing module (1), boundary extraction module (2), semantic information extraction module (3), mask fusion module (4), SAM optimization segmentation module (5); (1) Data preprocessing module This module filters the data, unifies the image size, and divides the data into training and testing sets; (2) Boundary extraction module Perform data enhancement on breast ultrasound images to extract corresponding structural information and boundary features; (3) Semantic Information Extraction Module Using a classification model, the classification model is trained through class-level labels to extract image semantic features and generate pseudo labels; (4) Mask fusion module A fusion strategy is applied to complementarily fuse the results of boundary extraction and semantic information extraction to form a preliminary segmentation mask; (5) SAM Optimization Segmentation Module The prompt coordinate information is generated according to the preliminary segmentation result, and the prompt information is used as the intermediate signal. The original image is segmented using SAM, and then post-processed to obtain the predicted segmentation result.

2. The weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimization segmentation according to claim 1 is characterized in that: In step 2, the method for data enhancement and boundary extraction of breast ultrasound images is as follows: 2.

1. Perform data enhancement on ultrasound images by contrast enhancement; 2.

2. Perform K-means clustering on the enhanced ultrasound image to achieve the division of target area and non-target area; 2.

3. Perform threshold segmentation on the clustered image to obtain a binary segmentation mask; 2.

4. Calculate the minimum enclosing rectangle of each suspected target area on the binary segmentation mask to determine the height and width, and then filter out the wrong area according to the aspect ratio to obtain the boundary extraction result.

3. The weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimization segmentation according to claim 1 is characterized in that: In step 3, the semantic information extraction method is specifically as follows: 3.

1. Classify and train the ultrasound image data using a convolutional neural network to obtain a trained neural network model, wherein the pre-trained convolutional neural network includes VGG, ResNet or DenseNet; 3.

2. Integrate CAM after the last convolutional layer of the convolutional neural network to highlight key features related to classification and generate a heat map of the breast ultrasound image, wherein the CAM method includes Grad-CAM, AblationCAM, LayerCAM, EigenGradCAM or Grad-CAM++; 3.

3. Convert the continuous-valued class activation map into a binary image using a threshold. For each generated class activation map, apply the threshold to the pixel values ​​of the activation map. The area with activation value higher than this threshold is considered as the effective activation area of ​​the category. Only pixels with activation value higher than the threshold are retained, and the rest are ignored to generate binary pseudo labels and obtain semantic information extraction results.

4. The weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimization segmentation according to claim 1 is characterized in that: In step 4, the segmentation mask fusion method is specifically as follows: 4.

1. For a breast ultrasound image, the set of all suspicious regions in the boundary extraction result is The set of all suspicious regions in the semantic information extraction result is 4.2 Calculation of L M With L C Intersection, if the intersection is not empty, take L M Zhong and L C The area with the largest intersection is taken as the fusion result, otherwise L C As a result of fusion.

5. The weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimization segmentation according to claim 1 is characterized in that: In step 5, the SAM optimization segmentation method is specifically as follows: 5.

1. For a breast ultrasound image, traverse all connected areas marked as 1 on the fused segmentation mask and find the largest connected area on the image; calculate the upper left corner coordinates and lower right corner coordinates of its minimum circumscribed rectangle to obtain location prompt information; 5.

2. The prompt information is sent to the prompt encoder of SAM as the seed region signal, the original ultrasound image is input to the image encoder, and the optimized segmentation result is obtained through the SAM segmentation model; 5.

3. Use morphological post-processing operations to fill the hole areas inside the target area in the SAM output results to obtain the final predicted segmentation results; the post-processing steps include at least one of conditional random field CRF, morphological operations or region growing algorithms.

6. The weakly supervised breast ultrasound image segmentation method based on strategy fusion and SAM optimization segmentation according to claim 1 is characterized in that: The weakly supervised learning strategies include at least one of segmentation mask fusion, classification training using image-level labels, preliminary positioning of the target area using CAM, and using the preliminary position information as an intermediate signal to be sent to SAM for segmentation.

Images