An interactive breast ultrasound tomographic image segmentation method and system based on initial mask and cross attention

Abstract

This invention discloses an interactive breast ultrasound tomography image segmentation method and system based on an initial mask and cross-attention, belonging to the field of medical image processing and computer-aided diagnosis technology. The method first uses a pre-trained automatic segmentation network to perform initial segmentation on the input breast ultrasound tomography image to obtain an initial mask. Then, a cross-attention fusion module is introduced into the interactive segmentation framework to effectively fuse the original image features, the initial mask, and user click information. Based on this, the segmentation result is continuously optimized through an iterative interactive mechanism, thereby achieving fine segmentation of the target region. Compared with existing technologies, this invention can fully utilize prior information from automatic segmentation, improve the initial accuracy of interactive segmentation, and enhance the fusion capability of multi-source information through the cross-attention mechanism. It significantly improves segmentation accuracy and stability while reducing the number of user interactions, and has good application prospects.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology, and in particular to an interactive breast ultrasound tomography image segmentation method and system based on initial masking and cross attention. Background Technology

[0002] Early screening and accurate diagnosis of breast cancer are crucial for reducing mortality. Ultrasound computed tomography (CT) technology, as an emerging medical imaging technique, shows promising application prospects in breast disease detection. Accurate segmentation of lesion areas in ultrasound tomographic images helps doctors assist in diagnosis and treatment planning.

[0003] In existing technologies, interactive image segmentation methods based on deep learning have been widely applied in the field of medical image processing. For example, by incorporating user click information and combining iterative optimization with neural network models, segmentation accuracy is gradually improved. However, these methods typically rely on multiple user interactions to correct the segmentation results, and lack effective prior information guidance in the initial stage, which can easily lead to problems such as inaccurate target region localization and slow convergence speed. Furthermore, existing methods often employ simple concatenation or weighting when fusing image features, user click information, and historical segmentation results, making it difficult to fully explore the correlations between multi-source information, thus limiting further improvements in segmentation performance.

[0004] Therefore, these existing technologies still have the following problems and shortcomings: (1) The lack of effective initial segmentation prior information leads to low segmentation accuracy under conditions of few interactions; (2) The efficiency of interactive information utilization is not high, and it requires a large number of clicks; (3) The multi-source information fusion method is relatively simple and it is difficult to achieve fine modeling of key areas, thus affecting the segmentation effect of complex lesion areas. Summary of the Invention

[0005] This invention provides an interactive breast ultrasound tomography image segmentation method and system based on initial masking and cross-attention. By introducing a pre-trained segmentation network to generate an initial mask, the accuracy of the initial segmentation stage is improved. Furthermore, by introducing a cross-attention fusion module into the interactive segmentation framework, the fusion capability of image features and user interaction information is enhanced. Through an iterative optimization mechanism, the model can obtain high-quality segmentation results with fewer interactions, thereby effectively reducing user operation costs and improving overall segmentation performance. This invention combines an efficient segmentation method with automatic segmentation priors and user interaction information, improving the segmentation accuracy of complex regions while reducing the number of user interactions. This addresses the problems of existing interactive medical image segmentation methods, such as the lack of effective initial segmentation results, insufficient multi-source information fusion capability, poor segmentation results in complex boundary regions, slow model iteration convergence speed, and the need for extensive manual interaction to obtain high-quality segmentation results.

[0006] According to the present invention, an interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention is provided, comprising the following steps: (1) Obtain the breast ultrasound tomography image to be segmented and the segmentation mask of the breast ultrasound tomography image to be segmented. (2) Input the breast ultrasound tomographic image with segmentation mask obtained in step (1) into the pre-trained automatic segmentation network to obtain the initial mask corresponding to the breast ultrasound tomographic image. The automatic segmentation network is an encoder-decoder structure network that introduces a joint spatial and channel modeling mechanism. The pre-trained automatic segmentation network is obtained by segmentation training of the breast ultrasound tomographic image and its segmentation mask obtained in step (1). (3) Introduce a cross-attention fusion module at the input end of the RITM interactive segmentation framework to construct the CAIM-IS network structure and obtain the interactive segmentation network model; Obtain the interactive input information of the breast ultrasound tomography image in step (1), and encode the interactive input information into a feature representation corresponding to the space of the breast ultrasound tomography image; fuse the breast ultrasound tomography image, the initial mask and the feature representation as the input of the interactive segmentation model; (4) Based on the input of step (3), the segmentation result of the breast ultrasound tomography image is obtained, and the segmentation result is used as the history mask for the next round of iteration until the iteration reaches the interaction number threshold or the segmentation result satisfies the cross-union ratio coefficient of 0.8 or above, thus obtaining the final segmentation result of the breast ultrasound tomography image.

[0007] Preferably, in step (2), the automatic segmentation network is the HSC-DSUNet network, which introduces spatial and channel attention mechanisms on the basis of the U-Net structure, and combines deep supervision strategy and ranking loss function to constrain features at different scales in order to improve the accuracy of the initial mask; The sorting loss function is defined as follows:

[0008] Where: F represents the foreground pixel sample set, that is, the set of all pixels in the lesion region of a breast ultrasound tomographic image, and i represents the index of the i-th foreground pixel in the foreground pixel set F. This represents the i-th foreground pixel. This indicates that the score of the breast ultrasound tomography image is higher than that of the foreground pixel. The number of background pixels, Represents pixels Relative ranking among all samples involved in the ranking To prevent extremely small constants with a denominator of zero.

[0009] Preferably, in step (3), the interactive input information includes foreground click points and background click points, and is encoded into an Euclidean distance map or probability distribution map through distance transformation, as an additional channel for the input of the interactive segmentation network model.

[0010] Preferably, in step (3), the cross-attention fusion module adopts a dual-branch structure, in which one branch is used to extract interactive information features and the other branch is used to extract joint features of the image and the mask, and cross-branch information guidance and feature reweighting are achieved through attention weights.

[0011] Preferably, in step (3), the cross-attention fusion module performs weighted modeling on different input features through a combination structure of channel attention and spatial attention, and achieves cross-feature weighted fusion through normalized weights, thereby enhancing the model's response capability to the foreground region.

[0012] Preferably, the threshold for the number of interactions is set to 20-25 times.

[0013] Preferably, the interactive segmentation network model is optimized using a normalized focus loss function to enhance its learning ability and improve segmentation accuracy and training stability.

[0014] Preferably, the crossover ratio coefficient reaches 0.95 or higher.

[0015] According to another aspect of the present invention, an interactive breast ultrasound tomography image segmentation system based on initial masking and cross-attention is provided, comprising: Breast ultrasound tomography image and its segmentation mask acquisition module: used to acquire the breast ultrasound tomography image to be segmented and the segmentation mask of the breast ultrasound tomography image to be segmented. The initial mask acquisition module corresponding to the breast ultrasound tomography image is used to input the obtained breast ultrasound tomography image with segmentation mask into a pre-trained automatic segmentation network to obtain the initial mask corresponding to the breast ultrasound tomography image. The automatic segmentation network is an encoder-decoder structure network that introduces a joint spatial and channel modeling mechanism. The pre-trained automatic segmentation network is obtained by segmentation training of the breast ultrasound tomography image and its segmentation mask. The input acquisition module of the interactive segmentation model is used to introduce a cross-attention fusion module at the input end of the RITM interactive segmentation framework, construct a CAIM-IS network structure, and obtain an interactive segmentation network model; acquire the interactive input information of the breast ultrasound tomography image, and encode the interactive input information into a feature representation corresponding to the space of the breast ultrasound tomography image; fuse the breast ultrasound tomography image, the initial mask, and the feature representation as the input of the interactive segmentation model; The breast ultrasound tomography image segmentation result acquisition module obtains the segmentation result of the breast ultrasound tomography image based on the input of the interactive segmentation model, and uses the segmentation result as the history mask for the next round of iteration until the number of iterations reaches the interaction threshold or the segmentation result satisfies the cross-union coefficient of 0.8 or higher, thus obtaining the final segmentation result of the breast ultrasound tomography image.

[0016] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages: (1) This invention improves the initial segmentation quality and reduces the burden of subsequent interactions. This invention introduces a pre-trained segmentation network to pre-segment breast ultrasound tomography images, generating an initial mask as prior input for interactive segmentation. Compared to existing technologies that start interaction directly with a blank mask or coarse initialization, this invention can provide more accurate target region localization results in the initial segmentation stage, effectively reducing initial errors and providing a reliable basis for subsequent interactive corrections, thereby significantly reducing the user's operational burden for the first interaction.

[0017] (2) This invention enhances the multi-source information fusion capability and improves interactive response efficiency. This invention introduces a cross-attention fusion module into the interactive segmentation framework, which can jointly model the original image features, user click prompts, and historical mask information, and adaptively allocate the importance of different information sources through an attention mechanism. Compared to traditional simple splicing or weighted fusion methods, this invention can more fully utilize user interaction intent, improve the model's response to click prompts, and make interaction correction more accurate and efficient.

[0018] (3) This invention improves the segmentation accuracy of complex regions and enhances the ability to characterize boundaries. In view of the characteristics of blurred lesion boundaries, complex tissue structures and low local contrast in breast ultrasound tomographic images, this invention uses the synergistic effect of initial mask prior and cross attention mechanism to enable the model to more accurately distinguish lesion areas from surrounding tissues and enhance the ability to identify small structures and complex boundaries, thereby effectively reducing undersegmentation, oversegmentation and missed segmentation, and improving the overall segmentation accuracy.

[0019] (4) This invention reduces the number of interactions and improves efficiency. This invention combines an iterative optimization mechanism, continuously updating the segmentation mask with the latest prediction results after each round of user interaction, enabling the model to quickly approximate the true target boundary. Compared with existing technologies, this invention requires fewer clicks to achieve the same segmentation accuracy, effectively shortening user annotation time and improving actual efficiency, making it particularly suitable for large-scale medical image annotation scenarios.

[0020] (5) This invention improves model stability and robustness. By fusing prior information from automatic segmentation with user interaction information, this invention enables the model to maintain relatively stable segmentation performance when faced with lesions of different sizes, shapes, and imaging qualities. Even in images with strong noise or unclear boundaries, reliable results can still be obtained through a small amount of interaction, demonstrating good generalization ability and robustness.

[0021] (6) This invention has both clinical application value and promotion prospects. This invention can effectively reduce the dependence on professional doctors' experience and time cost for fine annotation of medical images, improve the segmentation efficiency of breast ultrasound tomography images, and provide a reliable foundation for clinical auxiliary diagnosis, lesion volume measurement, three-dimensional reconstruction and subsequent intelligent analysis. It has good practical application value and industrial promotion prospects. Attached Figure Description

[0022] Figure 1 This invention relates to a breast ultrasound tomographic image and its corresponding mask image.

[0023] Figure 2 This is a schematic diagram of the pre-trained network (Hybrid Spatial-Channel Attention DeepSupervision-Guided UNet, HSC-DSUNet) involved in the present invention.

[0024] Figure 3 This invention relates to the Cross Attention Fusion Module (CAFM).

[0025] Figure 4This is a schematic diagram of the Cross-Attention InitialMask Interactive Segmentation (CAIM-IS) network structure.

[0026] Figure 5 This invention relates to a flowchart of all steps of interactive segmentation.

[0027] Figure 6 This is a comparison of the visualization results of the method proposed in this invention with other mainstream interactive segmentation methods.

[0028] Figure 7 This is a graph showing the relationship between the IoU value and the number of interactive clicks for the method proposed in this invention and other mainstream interactive segmentation methods. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0030] Example 1 This invention provides an interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention, the specific implementation process of which includes the following steps: (1) Acquiring breast ultrasound tomography image data and corresponding segmentation masks. The data used in this invention came from Hubei Provincial Cancer Hospital and Macheng People's Hospital in Hubei Province, and was acquired using breast ultrasound tomography equipment provided by Wuhan Vision Medical Imaging Co., Ltd. A total of 132 female patients aged 25 to 70 years participated in the data acquisition. These subjects were all diagnosed with breast lesions through routine breast ultrasound examination. The breast ultrasound tomography images were annotated under the guidance of experienced clinicians. For example... Figure 1As shown, breast ultrasound tomographic images and their corresponding labeled masks were obtained. The original images had blurred lesion boundaries and high noise levels, increasing the segmentation difficulty. Under the guidance of experienced clinicians, the breast ultrasound tomographic images were manually labeled to obtain corresponding segmentation masks. The dataset was divided according to the subject dimension, with a training set, validation set, and test set ratio of 7:1.5:1.5. The training set contained 843 images, the validation set contained 164 images, and the test set contained 213 images, providing data support for subsequent model training. Furthermore, to fully utilize the spatial correlation between adjacent image slices, this study adopted a 2.5D input method based on channel stitching. Specifically, three consecutive image slices were stitched together along the channel dimension to construct three-channel input data, thereby introducing the spatial information contained in adjacent slices while maintaining the two-dimensional convolutional network structure. Before inputting the data into the network, to reduce computational complexity and standardize data size, all images and their corresponding labels were uniformly adjusted to 512×512 pixels.

[0031] (2) Generate the initial segmentation mask. Input the preprocessed breast ultrasound tomography image into the pre-trained HSC-DSUNet network to obtain the initial mask of the breast ultrasound tomography image. The HSC-DSUNet is an improved U-Net structure, the specific structure of which is as follows: Figure 2 Specifically, as shown, a residual backbone network, ResNet34, is introduced in the encoding stage, combined with a spatial and channel joint modeling mechanism. Simultaneously, deep supervision and a ranking loss function are incorporated in the decoding stage to enhance the feature representation ability of lesion regions. The pre-training step involves separate automatic segmentation training of HSC-DSUNet on the same dataset. The ranking loss function is defined as follows:

[0032] Where: F represents the foreground pixel sample set, that is, the set of all pixels in the lesion region (foreground) of a breast ultrasound tomographic image, and i represents the index of the i-th foreground pixel in the foreground pixel set F. This represents the i-th foreground pixel. This indicates that the score of the breast ultrasound tomography image is higher than that of the foreground pixel. The number of background pixels, Represents pixels Relative ranking among all samples involved in the ranking To prevent extremely small constants with a denominator of zero.

[0033] (3) Constructing an interactive segmentation network. A cross-attention fusion module (CAFM) is introduced into the input of the RITM interactive segmentation framework to construct the CAIM-IS network structure, resulting in the interactive segmentation network model. The CAFM structure is as follows: Figure 3As shown, the CAIM-IS structure is as follows: Figure 4 As shown.

[0034] (4) Obtain interactive segmentation results. The original breast ultrasound tomographic image is concatenated with the initial mask obtained in step (2). Simultaneously, user click information on the image is received, and the click locations are encoded as distance maps or probability maps to maintain spatial consistency with image features. Then, the image, initial mask, and user click information are used as input, and feature fusion is performed through the CAFM module to obtain the current segmentation result. This result is then used as the historical mask for the next round of input, and iteratively updated in conjunction with new click information to gradually optimize the segmentation result. Specific iterative steps are as follows: Figure 5 As shown. The final segmentation result of the breast ultrasound tomography image is obtained when the number of iterations reaches the interaction threshold or the segmentation result satisfies the cross-union ratio (CUI) of 0.8 or higher.

[0035] (7) Output the segmentation results and evaluate the results. For example... Figure 6 As shown, the segmentation results of the method of this invention compared with other interactive segmentation methods under different numbers of clicks are as follows: from top to bottom, they are RITM-hrnet18s, FocalClick, MFP, CFR-ICL, and the CAIM-IS method proposed in this paper; from left to right, they are the original image, the ground truth annotation, and the segmentation results under 1, 2, 3, 5, and 10 clicks. This method can obtain relatively accurate segmentation boundaries with a small number of clicks, and the results are smoother and more complete.

[0036] Furthermore, during model training, a normalized focus loss function is used to optimize the network, enhancing its ability to focus on difficult-to-segment regions and improving overall segmentation performance. The normalized focus loss function is defined as follows:

[0037] in: This indicates the model's prediction results. Represents pixels The predicted probability at which This represents the normalization term for the weights of all pixels. These are adjustable parameters used to control the model's attention to difficult samples.

[0038] To further demonstrate the quantitative comparison between the proposed method and mainstream interactive segmentation methods, NoC (Number of Clicks) was used as the primary evaluation metric, combined with different accuracy thresholds, including NoC@80, NoC@85, and NoC@90. The NoC metric represents the average number of clicks required for the model to achieve a specified segmentation accuracy, and its definition is as follows:

[0039] in: Indicates the number of test samples. Indicates the first Each sample reaches the threshold in segmentation accuracy. The number of clicks required at that time. This article uses IoU as the accuracy measure. When the IoU value of the segmentation result reaches or exceeds a set threshold... When the time is reached, the sample is considered to have been segmented. Specifically: NoC@80: represents the average number of clicks required to achieve Dice ≥ 0.80; NoC@85: represents the average number of clicks required to achieve Dice ≥ 0.85; NoC@90: represents the average number of clicks required to achieve Dice ≥ 0.90. Table 1 shows a comparison of the quantitative results of the method proposed in this invention with mainstream interactive segmentation methods. Figure 7 The relationship between the number of clicks and segmentation accuracy of different methods is shown. It can be seen that the method of the present invention can achieve high segmentation accuracy with fewer clicks, demonstrating better interaction efficiency and segmentation performance.

[0040] Table 1

[0041] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention, characterized in that, Includes the following steps: (1) Obtain the breast ultrasound tomography image to be segmented and the segmentation mask of the breast ultrasound tomography image to be segmented. (2) Input the breast ultrasound tomographic image with segmentation mask obtained in step (1) into the pre-trained automatic segmentation network to obtain the initial mask corresponding to the breast ultrasound tomographic image. The automatic segmentation network is an encoder-decoder structure network that introduces a joint spatial and channel modeling mechanism. The pre-trained automatic segmentation network is obtained by segmentation training of the breast ultrasound tomographic image and its segmentation mask obtained in step (1). (3) Introduce a cross-attention fusion module at the input end of the RITM interactive segmentation framework to construct the CAIM-IS network structure and obtain the interactive segmentation network model; Obtain the interactive input information of the breast ultrasound tomography image in step (1), and encode the interactive input information into a feature representation corresponding to the space of the breast ultrasound tomography image; fuse the breast ultrasound tomography image, the initial mask and the feature representation as the input of the interactive segmentation model; (4) Based on the input of step (3), the segmentation result of the breast ultrasound tomography image is obtained, and the segmentation result is used as the history mask for the next round of iteration until the iteration reaches the interaction number threshold or the segmentation result satisfies the cross-union ratio coefficient of 0.8 or above, thus obtaining the final segmentation result of the breast ultrasound tomography image.

2. The interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention as described in claim 1, characterized in that, In step (2), the automatic segmentation network is the HSC-DSUNet network. This network introduces spatial and channel attention mechanisms on the basis of the U-Net structure, and combines deep supervision strategy and ranking loss function to constrain features at different scales in order to improve the accuracy of the initial mask. The sorting loss function is defined as follows: Where: F represents the foreground pixel sample set, that is, the set of all pixels in the lesion region of a breast ultrasound tomographic image, and i represents the index of the i-th foreground pixel in the foreground pixel set F. This represents the i-th foreground pixel. This indicates that the score of the breast ultrasound tomography image is higher than that of the foreground pixel. The number of background pixels, Represents pixels Relative ranking among all samples involved in the ranking To prevent extremely small constants with a denominator of zero.

3. The interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention as described in claim 1, characterized in that, In step (3), the interactive input information includes foreground click points and background click points, and is encoded into an Euclidean distance map or probability distribution map through distance transformation, serving as an additional channel for the input of the interactive segmentation network model.

4. The interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention as described in claim 1, characterized in that, In step (3), the cross-attention fusion module adopts a dual-branch structure, in which one branch is used to extract interactive information features and the other branch is used to extract joint features of the image and mask. Cross-branch information guidance and feature reweighting are achieved through attention weights.

5. The interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention as described in claim 1, characterized in that, In step (3), the cross-attention fusion module performs weighted modeling on different input features through a combination structure of channel attention and spatial attention, and achieves cross-feature weighted fusion through normalized weights, thereby enhancing the model's response to the foreground region.

6. The interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention as described in claim 1, characterized in that, The threshold for the number of interactions is set to 20-25 times.

7. The interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention as described in claim 1, characterized in that, The interactive segmentation network model is optimized using a normalized focus loss function to enhance its learning ability and improve segmentation accuracy and training stability.

8. The interactive breast ultrasound tomography image segmentation method based on initial masking and cross-attention as described in claim 1, characterized in that, The crossover ratio reaches 0.95 or higher.

9. An interactive breast ultrasound tomography image segmentation system based on initial masking and cross-attention, characterized in that, include: Breast ultrasound tomography image and its segmentation mask acquisition module: used to acquire the breast ultrasound tomography image to be segmented and the segmentation mask of the breast ultrasound tomography image to be segmented. The initial mask acquisition module corresponding to the breast ultrasound tomography image is used to input the obtained breast ultrasound tomography image with segmentation mask into a pre-trained automatic segmentation network to obtain the initial mask corresponding to the breast ultrasound tomography image. The automatic segmentation network is an encoder-decoder structure network that introduces a joint spatial and channel modeling mechanism. The pre-trained automatic segmentation network is obtained by segmentation training of the breast ultrasound tomography image and its segmentation mask. The input acquisition module of the interactive segmentation model is used to introduce a cross-attention fusion module at the input end of the RITM interactive segmentation framework, construct a CAIM-IS network structure, and obtain an interactive segmentation network model; acquire the interactive input information of the breast ultrasound tomography image, and encode the interactive input information into a feature representation corresponding to the space of the breast ultrasound tomography image; fuse the breast ultrasound tomography image, the initial mask, and the feature representation as the input of the interactive segmentation model; The breast ultrasound tomography image segmentation result acquisition module obtains the segmentation result of the breast ultrasound tomography image based on the input of the interactive segmentation model, and uses the segmentation result as the history mask for the next round of iteration until the number of iterations reaches the interaction threshold or the segmentation result satisfies the cross-union coefficient of 0.8 or higher, thus obtaining the final segmentation result of the breast ultrasound tomography image.

Images