Medical image segmentation method fusing text guidance and pseudo tag optimization

Abstract

The invention provides a medical image segmentation method fusing text guidance and pseudo tag optimization, and belongs to the technical field of medical image analysis. In view of the problem that accurate positioning based on semantic description is difficult to realize in a complex anatomical region due to the fact that a current medical image segmentation model generally lacks deep fusion of semantic information, firstly, logic relation cue words are designed for the shape, position and proximity relation of a target organ, and the understanding ability of the model for inter-organ structure semantics is enhanced; then, prompt words are coded by means of a CLIP pre-training model and fused into MedSAM for fine tuning, so that text information can guide positioning of a fuzzy boundary area more accurately; finally, a pseudo-label optimization strategy is introduced to effectively relieve the label noise problem caused by fuzzy edges, experiments are conducted on three public data sets AMOS22, BTCV and LiTS, the result shows that the method achieves comprehensive and advanced performance, and the excellent generalization ability and robustness of the method are fully verified.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis technology, specifically relating to a medical image segmentation method that integrates text guidance and pseudo-label optimization. Background Technology

[0002] Medical image segmentation, a core component of computer-aided diagnostic systems, aims to accurately divide specific anatomical structures or lesion regions within images, directly impacting the accuracy of clinical diagnosis and the reliability of treatment planning. However, medical images themselves are highly complex, with different tissue structures often exhibiting overlapping projections in imaging. Furthermore, significant feature differences exist between multimodal images such as CT and MRI, making high-precision annotation extremely difficult.

[0003] Currently, image annotation still heavily relies on manual drawing by professional physicians, which is not only time-consuming and labor-intensive but also suffers from significant inter-observer variability—different experts may only have 60%–70% agreement on the same image's annotation, severely limiting annotation efficiency and consistency. Therefore, developing automated, high-precision segmentation technologies is crucial for improving diagnostic efficiency, constructing high-quality annotated datasets, and promoting the implementation of precision medicine.

[0004] To reduce reliance on manual annotation, researchers have proposed methods such as automatic annotation, semi-supervised learning, and transfer learning, aiming to expand the applicability of models using limited labeled data. However, these methods still face a series of challenges in practical applications: the quality of automatic annotation is difficult to guarantee, semi-supervised methods have limited generalization ability to out-of-distribution data, and the adaptation effect of transfer learning varies across different modalities. With technological evolution, unsupervised learning has been highly anticipated, offering new possibilities for achieving segmentation with less manual intervention. However, in medical images, the boundaries of certain organs or lesions are often blurred, their shapes are irregular, and their internal textures are highly heterogeneous, making it difficult for unsupervised models to achieve accurate boundary segmentation in these challenging areas, thus limiting their clinical applicability.

[0005] Against this backdrop, the MedSAM model represents a significant breakthrough in medical image segmentation, successfully introducing the capabilities of a general-purpose visual foundation model into medical image analysis. Trained on a massive multimodal medical dataset, this model demonstrates exceptional generalization performance: it achieves high-precision segmentation across various imaging modalities, including CT and MRI, and adapts to complex variations in anatomical structures and lesion morphologies. The introduction of MedSAM marks the first validation of a single model's segmentation capabilities across modalities, organs, and lesions, providing crucial technical support and a practical paradigm for building a generalized and robust medical image segmentation foundation model. It also lays a solid foundation for future intelligent segmentation applications in scenarios with minimal or even zero annotation.

[0006] However, current MedSAM-based medical image segmentation models generally lack deep fusion of semantic information, making it difficult to achieve accurate localization based on semantic descriptions in complex anatomical regions. Introducing text-guided mechanisms and leveraging natural language understanding to enhance semantic constraints on target organs has become an important research direction for improving the discriminative capabilities of MedSAM models. Meanwhile, in imaging such as abdominal CT scans, due to limited grayscale differences between adjacent organs and frequent interference from imaging noise, segmentation of regions with blurred boundaries still faces significant challenges, easily leading to misclassification and omission of edge regions. This problem has become a core bottleneck restricting the application value of models in real-world clinical scenarios and urgently needs to be addressed through novel segmentation strategies combining semantic guidance and local perception. Summary of the Invention

[0007] To address the problem that current MedSAM-based medical image segmentation models generally lack deep integration of semantic information, making it difficult to achieve accurate localization based on semantic description in complex anatomical regions, this invention proposes a medical image segmentation method that integrates text guidance and pseudo-label optimization.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A medical image segmentation method integrating text guidance and pseudo-label optimization is proposed. Based on the MedSAM large model, firstly, logical relationship prompts are designed based on the shape, location, and proximity of the target organ to enhance the model's understanding of the structural semantics between organs. Subsequently, the prompts are encoded using a CLIP pre-trained model and integrated into MedSAM for model fine-tuning, enabling text information to more accurately guide the localization of blurred boundary regions. Finally, a pseudo-label optimization strategy is introduced to effectively alleviate the label noise problem caused by blurred edges, thereby improving the quality of pseudo-labels and segmentation robustness.

[0010] As a preferred embodiment of the present invention, the text-guided segmentation process first targets the target organ. Defined feature tuples Then through a template function , feature tuple Combined into a complete and logical text prompt Then, based on the CLIP model, the paired image-text data are... They are mapped to the same semantic feature space to maximize the similarity of their feature vectors, and finally the corresponding images are sent to the mask decoder.

[0011] As a preferred embodiment of the present invention, the pseudo-label optimization process targets the organ. The predicted high-confidence samples generate pseudo-labels x. Simultaneously, two entropy maps are generated for the pseudo-labels x: an unenhanced entropy map Y(x) and an enhanced entropy map Y(X). The two entropy maps are weighted and combined to generate a new selective entropy mask M. Then, a false negative region mask N is generated based on the predicted probability p(x). Subsequently, by performing an element-wise logical AND operation between the false negative region mask N and the selective entropy mask M, regions with high uncertainty and potential false negatives are further filtered out. Finally, the model is optimized based on the loss function.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0013] 1) To evaluate the effectiveness of the text-guided pseudo-label optimization method proposed in this invention, qualitative and quantitative analyses were conducted on the original training model, the training model with added multi-target interactive prompts, and the training model with added pseudo-label optimization. Experimental results show that removing the prompt module leads to a 2.08% decrease in the Dice coefficient, and irregular segmentation and organ misclassification occur in blurred boundary regions; disabling pseudo-label optimization reduces accuracy by 2.15%, and generates more discrete errors and continuity loss in low-contrast regions. Quantitative and qualitative analyses jointly verify the effectiveness of each module and their complementary relationship, providing an important basis for model optimization.

[0014] 2) To comprehensively evaluate the performance of the proposed method, comparative experiments were conducted on three publicly available datasets: AMOS22, BTCV, and LiTS, against various state-of-the-art segmentation models. Experimental results show that the proposed method achieves superior performance on all three authoritative datasets (AMOS22, BTCV, and LiTS), fully validating its excellent generalization ability and robustness.

[0015] 3) In multiple organ segmentation tasks, this invention significantly outperforms all baseline models in the three core metrics of Dice, mIoU and ASSD, especially in boundary segmentation accuracy (ASSD is greatly reduced), demonstrating better detail fitting and boundary adaptation capabilities.

[0016] 4) To verify the effectiveness of the interactive prompting strategy in this model, a comparative experiment was conducted with methods based on non-interactive prompts. Experimental results show that in regions with blurred edges or unclear structures, the method of this invention significantly outperforms the benchmark in terms of segmentation accuracy. Quantitative analysis shows that the model using interactive prompts achieves a higher Dice coefficient, indicating better consistency between its segmentation results and the actual annotations, thus proving the effectiveness of interactive prompts in improving segmentation performance. Attached Figure Description

[0017] Figure 1This is a construction diagram of the model (Prompt Pseudo label MedSAM) proposed in this invention, which combines text guidance and pseudo-label optimization based on the MedSAM large model.

[0018] Figure 2 The original training model and the training model after adding multi-target interactive prompts and pseudo-label optimization methods are analyzed and compared, and compared with the real labels.

[0019] Figure 3 This is an analysis and comparison of training models with and without adding multi-target interactive prompts under the same label (Dice coefficient). Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0021] Example 1

[0022] Please see Figure 1 As shown, this invention proposes a method combining text guidance and pseudo-label optimization based on a MedSAM large-scale model. First, logical relationship prompts are designed based on the shape, location, and proximity relationships of the target organ to enhance the model's understanding of the structural semantics between organs. Then, the prompts are encoded using a CLIP pre-trained model and integrated into MedSAM for model fine-tuning, enabling the text information to more accurately guide the localization of ambiguous boundary regions. Finally, a pseudo-label optimization strategy is introduced to effectively alleviate label noise caused by ambiguous edges, thereby improving pseudo-label quality and segmentation robustness.

[0023] The above method, which combines text guidance and pseudo-label optimization based on the MedSAM large model, can be divided into two stages:

[0024] (1) Text-guided segmentation process, which first targets the target organ Defined feature tuples Then through a template function , feature tuple Combined into a complete and logical text prompt Then, based on the CLIP model, the paired image-text data are... They are mapped to the same semantic feature space to maximize the similarity of their feature vectors, and finally, the corresponding images are fed into the mask decoder. The specific process is as follows:

[0025] First, identify the target organ based on the text information. This transforms unstructured medical knowledge into structured text descriptions. For a target organ... Define its prior knowledge as a feature tuple:

[0026] (1)

[0027] Each component is a textual description of the corresponding feature.

[0028] Through a template function , feature tuple Combined into a complete and logical text prompt :

[0029] (2)

[0030] Secondly, the CLIP model includes a text encoder. and image encoder The goal is to pair image-text data. Mapping them to the same semantic feature space maximizes the similarity of their feature vectors.

[0031] (3)

[0032] in, It is cosine similarity. It's a temperature parameter. It is a collection of text in a batch.

[0033] because It contains richer semantic information This allows the model to maximize At that time, it forces the image encoder Visual features that match prior knowledge such as shape, position, and boundaries must be extracted during training to improve image quality. (Including organs) It provides positive sample text with more information and more precise semantics. .

[0034] (2) The pseudo-label optimization process for target organs The predicted high-confidence samples generate pseudo-labels x. Simultaneously, two entropy maps are generated for each pseudo-label x: an unenhanced entropy map Y(x) and an enhanced entropy map Y(X). These two entropy maps are weighted and combined to generate a new selective entropy mask M. Then, a false negative region mask N is generated based on the predicted probability p(x). Subsequently, by performing an element-wise logical AND operation between the false negative region mask N and the selective entropy mask M, regions with high uncertainty and potential false negatives are further filtered out. Finally, the model is optimized based on the loss function. The specific process is as follows:

[0035] First, the model will target the organ. The predicted high-confidence samples generate pseudo-labels x for subsequent training. However, pseudo-labels may contain errors, especially false negatives. To reduce false negatives and further improve the quality of pseudo-labels, two entropy maps are generated: an unenhanced entropy map Y(x) and an enhanced entropy map Y(X). The unenhanced entropy map Y(x) and the enhanced entropy map Y(X) are weighted and combined to generate a new selective entropy mask M.

[0036] (4)

[0037] Secondly, the selective entropy mask M is used to identify high-uncertainty regions (i.e., regions where the model may make mistakes), providing a basis for subsequent suppression of false negatives in pseudo-labels. Then, a false negative region mask N is generated based on the predicted probability p(x), where:

[0038] (5)

[0039] Finally, by performing an element-wise logical AND operation between the false negative region mask N and the selective entropy mask M, regions with high uncertainty and potential false negatives are further filtered out.

[0040] Example 2

[0041] Model (Prompt Pseudo label MedSAM) performance evaluation experiment:

[0042] 1) Ablation Experiment: In order to evaluate the effectiveness of the text-guided pseudo-label optimization method proposed in this invention, qualitative and quantitative analyses were performed on the original training model, the training model with added multi-target interactive prompts, and the training model with added pseudo-label optimization method.

[0043] Table 1 Qualitative and Quantitative Analysis

[0044] LiTS dataset Dice(↑) ASSD(↓) mIoU(↑) Ours 96.58 0.11 96.29 Optimization methods for removing pseudo-tags 94.43 0.16 95.10 Remove prompt word module 94.5 0.16 94.26 Original model 93.93 0.23 93.59

[0045] The contributions of key model components were evaluated through systematic ablation experiments. After sequentially removing the multi-target interactive prompt module and the pseudo-label optimization method, the experimental results showed that removing the prompt module led to a 2.08% decrease in the Dice coefficient, resulting in irregular segmentation and organ misclassification in blurred boundary regions; disabling pseudo-label optimization reduced Dice accuracy by 2.15%, producing more discrete errors and continuity loss in low-contrast regions. Figure 2 (As shown in Table 1). Quantitative and qualitative analyses jointly verified the effectiveness of each module and their complementary relationship, providing an important basis for model optimization.

[0046] 2) Comparative Experiments: To comprehensively evaluate the performance of the proposed method, comparative experiments were conducted on three public datasets: AMOS22, BTCV, and LiTS, against various state-of-the-art segmentation models. These are shown in Tables 2, 3, and 4.

[0047] Table 2 Comparison results on the AMOS22 dataset

[0048] AMOS22 dataset Dice(↑) ASSD(↓) mIoU(↑) VA-TransUNet 87.34 0.42 87.03 DeepLabV3+ 87.52 0.21 86.8 Continual Learning 88.8 0.26 87.32 nnUnet 89.5 0.17 86.53 CLIP-driven 90.88 0.16 91.57 Ours 97.57 0.08 97.4

[0049] Table 3 Comparison results on the BTCV dataset

[0050] BTCV dataset Dice(↑) ASSD(↓) mIoU(↑) VA-TransUNet 86.89 0.36 87.02 DeepLabV3+ 85.66 0.41 83.7 Continual Learning 87.5 0.31 88.07 nnUnet 88.52 0.29 86.79 CLIP-driven 90.56 0.21 89.98 Ours 91.3 0.09 92.07

[0051] Table 4 Comparison results on the LiTS dataset

[0052] LiTS dataset Dice(↑) ASSD(↓) mIoU(↑) VA-TransUNet 92.25 0.25 90.17 DeepLabV3+ 89.8 0.35 88.31 Continual Learning 90.22 0.31 89.1 nnUnet 91.57 0.24 90.35 CLIP-driven 94.02 0.15 93.11 Ours 96.58 0.11 96.29

[0053] The results show that the present invention achieves comprehensive leading performance on three authoritative datasets: AMOS22, BTCV, and LiTS, fully validating its excellent generalization ability and robustness. In multiple organ segmentation tasks, the present invention significantly outperforms all baseline models in the three core metrics of Dice, mIoU, and ASSD, especially in boundary segmentation accuracy (with a significant reduction in ASSD), demonstrating superior detail fitting and boundary adaptation capabilities.

[0054] 3) Parameter sensitivity analysis:

[0055] To verify the effectiveness of the interactive prompting strategy in this model, a comparative experiment was conducted with a method based on non-interactive prompts. Experimental results show that in regions with blurred edges or unclear structures, the method of this invention significantly outperforms the benchmark in terms of segmentation accuracy. Quantitative analysis shows that the model using interactive prompts achieves a higher Dice coefficient (…). Figure 3 As shown in the figure, the segmentation results have better consistency with the actual annotations, thus proving the effectiveness of interactive prompts in improving segmentation performance.

[0056] The above datasets and models are sourced from:

[0057] AMOS22: The AMOS22 dataset is a leading benchmark dataset for multi-organ tumor analysis in the field of medical image segmentation. It was constructed by Johns Hopkins University in collaboration with several top medical centers worldwide. It was also the dataset used in the MICCAI2022 International Challenge for Multimodal Abdominal Segmentation. Primarily designed to advance medical image segmentation techniques, it collects abdominal CT and MRI scans from real-world clinical scenarios. Each case includes voxel-level annotations for 15 abdominal organs, making it a comprehensive benchmark dataset. (https: / / amos22.grand-challenge.org / )

[0058] BTCV: The BTCV dataset is one of the authoritative benchmark datasets in the field of medical image segmentation, focusing on the accurate analysis of multiple abdominal organs. It was jointly constructed by internationally renowned research institutions and multiple medical centers. This dataset includes 50 high-quality abdominal CT scans. All samples were acquired during the portal venous phase with enhanced scanning sequences, with slice spacing controlled within the range of 1-5mm and a uniform image resolution of 512×512 pixels, realistically reproducing the equipment differences encountered in clinical practice. The dataset fully annotates 13 key abdominal organs (including spleen, both kidneys, gallbladder, esophagus, liver, stomach, aorta, inferior vena cava, portal vein and splenic vein, pancreas, and bilateral adrenal glands). Each case underwent a tiered quality control process: initial annotation by two experienced undergraduate students, followed by final confirmation by radiology experts based on volumetric verification, resulting in pixel-level accurate gold-standard segmentation masks. Furthermore, this dataset particularly emphasizes the depiction of complex anatomical relationships, covering organ morphological variations in patients of different body types, as well as cases of blurred organ boundaries due to individual differences, fully simulating the segmentation challenges in real clinical scenarios. (https: / / www.synapse.org / #!Synapse:syn3193805 / wiki / ).

[0059] LiTS: The LiTS dataset is one of the most authoritative benchmark datasets for liver tumor analysis in the field of medical image segmentation, jointly released by the Technical University of Munich and several medical institutions. This dataset contains 131 training samples and 70 test samples, all of which are enhanced CT scan images from the venous phase, with slice spacing of 1-5 mm and a uniform resolution of 512×512 pixels. It covers different scanning devices (GE, Siemens, etc.) and various pathological types (including hepatocellular carcinoma, metastatic tumors, etc.). The dataset provides pixel-level gold-standard segmentation masks for liver parenchyma and tumor regions, jointly annotated by at least three senior radiologists. Tumor sizes vary significantly (diameter 3mm-200mm), fully reflecting the complexities of real-world clinical scenarios. Particularly noteworthy is the inclusion of annotations for a large number of small tumors (<10mm) and cases with blurred boundaries, making it an ideal testing platform for validating algorithm robustness. As the official dataset for the MICCAI 2017 challenge, LiTS 2017 has driven the development of several groundbreaking algorithms, including nnUNet and 3D U-Net, and remains one of the most widely used benchmark datasets in liver imaging AI research. (https: / / competitions.codalab.org / competitions / 17094).

[0060] VA-TransUNet is one of the first models to integrate Transformers into medical image analysis. This method leverages the high-resolution spatial details of the U-Net encoder while taking advantage of Transformers' strengths in global context modeling, which is crucial in medical image segmentation. (https: / / github.com / BeautySilly / VA-TransUNet).

[0061] DeepLabV3+ is a semantic segmentation network proposed by Google. It combines the advantages of both encoder-decoder and spatial pyramid pooling (SPP) architectures to simultaneously acquire multi-scale contextual information and fine-grained target boundaries. (https: / / github.com / VainF / DeepLabV3Plus-Pytorch?tab=readme-ov-file#31-visualize-training-optional)

[0062] ContinualLearning is an innovative model that solves a core challenge in medical image segmentation. Utilizing pseudo-labels and a CLIP-driven dynamic head architecture, the model's effectiveness has been validated on multiple abdominal CT datasets. (https: / / github.com / MrGiovanni / ContinualLearning).

[0063] nnUnet: nnU-Net is a medical image segmentation framework that adapts to any new dataset, proposed by researchers (Fabian Isensee, Jens Petersen, and Andre Klein) from the German Cancer Research Center, Heidelberg University, and Heidelberg University Hospital. This framework automatically adjusts all hyperparameters based on the properties of a given dataset, without requiring manual intervention. (https: / / github.com / MIC-DKFZ / nnUNet).

[0064] CLIP-driven: The CLIP-Driven Universal Model is a universal model based on CLIP-learned text embeddings, which is then incorporated into a segmentation model. This CLIP-based label encoding captures anatomical relationships, enabling the model to learn structured feature embeddings and segment 25 organs and 6 types of tumors. The proposed model was developed from a combination of 14 datasets and trained using a total of 3,410 CT scans. (https: / / github.com / ljwztc / CLIP-Driven-Universal-Model).

Claims

1. A medical image segmentation method fusing text guidance and pseudo label optimization, characterized in that, Based on the MedSAM large model construction, first, according to the shape, position and adjacent relationship of the target organ, the logical relationship prompt words are designed to enhance the model's understanding of the semantic structure between organs; then, with the help of the CLIP pre-training model, the prompt words are encoded and integrated into the MedSAM for model fine-tuning, so that the text information can more accurately guide the positioning of the fuzzy boundary area; finally, the pseudo-label optimization strategy is introduced to effectively alleviate the label noise problem caused by the fuzzy edge, thereby improving the quality of the pseudo-label and the segmentation robustness.

2. The method of fusing text guidance and pseudo-label optimization for medical image segmentation of claim 1, wherein, The text-guided segmentation process first targets the target organ defined feature tuples , and then through a template function , the feature tuples are synthesized into a complete, logical text prompt , and then according to the CLIP model, the paired image-text data are mapped into the same semantic feature space, so that their feature vectors have the highest similarity, and finally the corresponding image is sent into the mask decoder.

3. The method of fusing text guidance and pseudo-label optimization for medical image segmentation of claim 2, wherein, The specific process of text-guided segmentation is as follows: First, the target organ is identified from the textual information unstructured medical knowledge into structured textual descriptions; for a target organ define its prior knowledge as a feature tuple: （1） where each component is a text description of the corresponding feature; Through a template function , the feature tuples are synthesized into a complete, logical text prompt : （2） Second, the CLIP model contains a text encoder and an image encoder with the goal of mapping paired image-text data into the same semantic feature space, so that their feature vectors have the highest degree of similarity; （3） wherein, is a cosine similarity, is a temperature parameter, is a collection of texts in a batch; Due to the richer semantic information , the model is forced to maximize the likelihood of the image encoder to extract visual features that match the prior knowledge of shape, location, boundary, etc., thus providing the image (containing organs ) with more informative and semantically accurate positive sample texts during training .

4. The method of fusing text guidance and pseudo-label optimization for medical image segmentation of claim 3, wherein, Pseudo-label optimization process for target organs The predicted high-confidence samples generate pseudo-labels x, while generating two entropy maps Y(x) and Y(x) for the pseudo-labels x, one without enhancement and one with enhancement, combining the two entropy maps with a weight, generating a new selective entropy mask M, and then generating a false negative region mask N according to the prediction probability p(x), then, through the element-wise logical AND operation between the false negative region mask N and the selective entropy mask M, further filtering out the regions with high uncertainty and possibly false negatives, and finally optimizing the model according to the loss function.

5. The method of fusing text guidance and pseudo-label optimization for medical image segmentation of claim 4, wherein, The specific process of pseudo-label optimization is as follows: First, the model will target the organ The predicted high-confidence samples generate pseudo-labels x for subsequent training, but the pseudo-labels may be wrong, especially false negatives. In order to reduce false negatives and further improve the quality of pseudo-labels, two entropy maps, Y(x) without enhancement and Y(X) with enhancement, are generated. The two entropy maps are combined by weighting to generate a new selective entropy mask M: （4） Secondly, the selective entropy mask M is used to identify high-uncertainty areas (i.e. areas where the model may make mistakes), which provides a basis for subsequent suppression of false negatives in the pseudo-label. Then, according to the predicted probability p(x), the false negative area mask N is generated, where: （5） Finally, by performing element-wise logical AND (AND) operation between the false negative area mask N and the selective entropy mask M, the high-uncertainty and possibly false negative areas are further filtered out.

Images