Medical image segmentation method based on SAM multi-modal multi-stage fusion semantic prompt

Abstract

The present application relates to the field of image processing, and proposes a kind of medical image segmentation method based on SAM multi-modal multi-stage fusion semantic prompt, obtains multi-modal target medical image data and carries out feature extraction and multi-modal feature fusion, to obtain unified semantic feature embedding;Difficult to distinguish negative sample is constructed and is subjected to differential feature loss optimization processing;According to unified semantic feature embedding, medical image segmentation is carried out to obtain final segmentation mask.The present application avoids the influence of complex medical scene and multi-modal data, improves the accuracy and stability of segmentation.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a medical image segmentation method based on SAM (Multimodal Multi-stage Fusion Semantic Cueing). Background Technology

[0002] Accurate and reliable medical image segmentation technology plays an irreplaceable role in key scenarios such as tumor screening, lesion localization, and anatomical structure analysis. Traditional U-Net-based segmentation methods have achieved excellent segmentation accuracy under controlled experimental conditions, but due to technical limitations, their adaptability to clinical applications remains significantly limited, making it difficult to meet the actual needs of diverse clinical scenarios. This type of framework exhibits strong modality dependence; its advantages are mainly concentrated in high-contrast, low-noise imaging modalities (such as CT and MRI), where it can accurately capture clear anatomical boundaries and lesion features. However, in low-contrast, noise-sensitive, and unstable imaging modalities (such as ultrasound), its segmentation performance deteriorates significantly, failing to effectively distinguish lesions from surrounding normal tissue, making it difficult to adapt to the imaging needs of primary healthcare institutions or complex clinical scenarios. Furthermore, the training of such frameworks heavily relies on large-scale, pixel-level precise segmentation annotations. However, medical image annotation work requires professional physicians to spend a lot of time and effort to complete, making it difficult to efficiently obtain large-scale annotated data in actual clinical environments. This pain point severely restricts the scalability and universality of such methods in different clinical scenarios, and makes it impossible to achieve widespread application across institutions and modalities.

[0003] To overcome the limitations of traditional segmentation methods, basic segmentation models, represented by SAM (Segment Anything Model), have emerged. These models achieve interactive segmentation based on geometric cues (such as points and boxes), reducing reliance on large-scale labeled data to some extent. However, natural images and medical images differ significantly in imaging mechanisms, feature distribution, and semantic meaning—a "domain gap" between the natural and medical domains. This means that features learned from natural images during model training cannot be effectively transferred to medical scenarios, making it difficult to guarantee segmentation accuracy. Biomedical-specific segmentation methods such as BiomedCLIP, MedSAM, and TextSAM-EUS attempt to integrate text guidance into SAM models to achieve accurate segmentation of medical images, but these works still have significant technical shortcomings. First, the prompt generation method is simplistic, relying solely on text prompts for segmentation guidance and lacking integration with medical image features. This results in the generated text prompts being disconnected from the actual image features, failing to accurately guide the model to locate key regions.

[0004] Second, the cross-modal semantic alignment effect is poor. The core reason is that it relies on a fixed pre-trained text encoder and does not conduct joint cross-modal adaptation training for the characteristics of medical images and clinical texts. The training data of the general pre-trained text encoder is mainly general text, which cannot fully adapt to the professionalism and rigor of medical terminology. Furthermore, it does not learn the relationship between medical image features and clinical text semantics at the same time, resulting in poor semantic alignment between text semantics and medical image features.

[0005] Third, the models lack adaptability to complex medical scenarios and struggle to handle intricate clinical situations. In real-world clinical settings, there are significant individual differences between cases and interference from tissues with highly similar anatomical structures. Existing models lack the ability to accurately identify subtle features, cannot achieve precise semantic matching and boundary segmentation of key structures, and are prone to missed or incorrect segmentation results, failing to meet the segmentation accuracy requirements for clinical diagnosis and treatment planning.

[0006] Therefore, designing a medical image segmentation method to avoid the influence of complex medical scenarios and multimodal data, and to improve the accuracy and stability of segmentation, has become an urgent problem to be solved. Summary of the Invention

[0007] Based on this, this invention proposes a medical image segmentation method based on SAM (Semantic Awareness and Analysis) with multimodal and multi-stage fusion semantic prompts. Through multimodal feature fusion at multiple stages, it achieves deep semantic fusion of medical images and text descriptions, completing accurate semantic alignment and cross-modal deep fusion of the two types of features, eliminating the semantic gap between modalities, and providing high-quality feature support for subsequent segmentation tasks. Furthermore, by generating difficult-to-distinguish negative samples and optimizing the loss, a high-quality, highly targeted set of semantic comparison samples is constructed, providing core support for subsequent comparative learning and guiding the model to deeply learn the semantic consistency between regional features and correct text descriptions. Even under interference from highly similar negative samples, it forces the model to accurately distinguish the semantic differences between positive and negative samples, significantly improving the model's discrimination accuracy and anti-interference ability in fine-grained medical semantic scenarios, further optimizing feature learning effects. Moreover, through semantic prompts that precisely guide the segmentation process, SAM can fully combine the visual features of medical images with the semantic information of clinical text, achieving accurate segmentation of lesion regions and anatomical structures in medical images. This effectively solves the segmentation deviation and boundary ambiguity problems caused by insufficient semantic information in traditional geometric prompts. This invention improves the accuracy and stability of segmentation.

[0008] This invention proposes a medical image segmentation method based on SAM (Multimodal Multi-Stage Fusion Semantic Prompt), comprising: Multimodal target medical image data is acquired and feature extraction and multimodal feature fusion are performed to obtain unified semantic feature embedding. The feature extraction includes image feature extraction and text feature extraction. The multimodal feature fusion is based on self-attention modeling and bidirectional cross-attention. The unified semantic feature embedding is based on multi-stage feature encoding and fusion processing. Difficult-to-distinguish negative samples are constructed and differential feature loss is optimized. The difficult-to-distinguish negative samples are generated based on a large language model, and the differential feature loss optimization is based on dynamic weights and regularization constraints. Medical image segmentation is performed based on unified semantic feature embedding to obtain a final segmentation mask. The medical image segmentation is based on a lightweight adapter, an image encoder, a mask decoder, and a low-rank adaptation module.

[0009] Furthermore, the step of acquiring multimodal target medical image data and performing feature extraction and multimodal feature fusion to obtain unified semantic feature embedding specifically includes: Multimodal target medical image data is used as input images and divided into multiple fixed-size patch blocks. Initial visual features are obtained through linear mapping. Initial visual features are input into an image feature extraction network, which is based on four hierarchical stages. In each stage, features are modeled through multiple Swing Transformer blocks. Each Swing Transformer block alternately employs different multi-head attention mechanisms, including window multi-head self-attention and shifted window multi-head self-attention. The specific algorithm of the Swing Transformer block is as follows: , in, This indicates the output characteristics of the Swing Transformer block. This indicates a multi-head attention mechanism. This indicates the output characteristics of the previous Swing Transformer block. l Presentation layer index; At the end of each stage, feature downsampling is performed based on patch block stitching to obtain image features for the four stages; Text feature extraction is performed based on a staged encoder, which is based on the PubMedBERT encoder. The 12 Transformer layers in the staged encoder are divided into four stages: layers 1 to 6 are stage 1, layers 7 to 8 are stage 2, layers 9 to 10 are stage 3, and layers 11 to 12 are stage 4, thus obtaining text features for the four stages. Self-attention modeling is performed on the image features and text features at each stage to obtain self-attention-enhanced image features and self-attention-enhanced text features. Then, cross-modal information interaction is performed according to a bidirectional cross-attention mechanism. The cross-modal information interaction includes vision-to-text fusion and text-to-vision fusion. The specific algorithm for the cross-modal information interaction is as follows: , , in, This represents the visual-to-text fusion feature. This represents the text-to-visual fusion feature. This indicates self-attention enhanced text features. This represents self-attention enhanced image features. and These represent text multilayer perceptron and image multilayer perceptron, respectively. This represents a bidirectional cross-attention mechanism; Visual-to-text fusion features and text-to-visual fusion features are used as inputs to the next stage of text encoding and image encoding to obtain a unified semantic feature embedding for the four-stage encoding fusion.

[0010] Furthermore, the step of constructing indistinguishable negative samples and performing differential feature loss optimization specifically includes: Based on a large language model, multiple indistinguishable negative samples are generated for each training sample. The indistinguishable negative samples are semantically similar to the training samples but differ in key medical meanings. Text samples that are difficult to distinguish from negative samples are encoded and embedded to obtain their features. Positive sample text samples are also encoded and embedded to obtain their features. The similarity between the image features and the features of both positive and negative samples is calculated. The specific algorithm for this similarity calculation is as follows: , , in, and These represent the similarity between image features and positive sample text features, and between negative sample text features that are difficult to distinguish, respectively. This represents the features of the transposed image. This represents the features of positive sample text. This indicates that it is difficult to distinguish negative sample text features. Indicates the temperature coefficient; A contrastive learning loss function for hard-to-distinguish negative samples is constructed and optimized using differential feature loss. The specific algorithm for the contrastive learning loss function for hard-to-distinguish negative samples is as follows: , , in, This indicates a contrastive learning loss that makes it difficult to distinguish between negative samples. Indicates the number of positive samples. Indicates the ordinal number of the positive sample. This represents the regularization coefficient for positive samples, and K represents the number of negative samples that are difficult to distinguish. This indicates that the ordinal numbers of negative samples are difficult to distinguish. This indicates that it is difficult to distinguish the weights of negative samples. Representing image features and the first The similarity of negative sample text features is difficult to distinguish. This represents the weight hyperparameter.

[0011] Furthermore, the step of performing medical image segmentation based on unified semantic feature embedding to obtain the final segmentation mask specifically includes: Unified semantic features are embedded into the input SAM model, and semantic cues are obtained based on a lightweight adapter. The lightweight adapter is based on a two-layer multilayer perceptron with a GELU activation function. The specific algorithm for the semantic cues is as follows: , in, Indicates semantic hints, Indicates a lightweight adapter. Represents unified semantic feature embedding; The image encoder obtains the image encoding, and then the image encoding and semantic prompts are input into the mask decoder. Based on the cross-modal attention mechanism, the image encoding is guided by the semantic prompts to obtain the final segmentation mask.

[0012] This invention proposes a medical image segmentation system based on SAM (Multimodal Multi-Stage Fusion Semantic Prompt), comprising: The feature extraction module is used to acquire multimodal target medical image data and perform feature extraction and multimodal feature fusion to obtain unified semantic feature embedding. The feature extraction includes image feature extraction and text feature extraction. The multimodal feature fusion is based on self-attention modeling and bidirectional cross-attention. The unified semantic feature embedding is based on multi-stage feature encoding and fusion processing. The negative sample loss optimization module is used to construct hard-to-distinguish negative samples and perform differential feature loss optimization processing. The hard-to-distinguish negative samples are generated based on a large language model, and the differential feature loss optimization processing is based on dynamic weights and regularization constraints. The segmentation module is used to segment medical images based on unified semantic feature embedding to obtain a final segmentation mask. The medical image segmentation is based on a lightweight adapter, an image encoder, a mask decoder, and a low-rank adaptation module.

[0013] The present invention also provides a storage medium that stores one or more programs, which, when executed by a processor, implement the medical image segmentation method based on SAM multimodal multi-stage fusion semantic prompting as described above.

[0014] The present invention also provides a computer device, the computer device including a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the medical image segmentation method based on SAM with multimodal and multi-stage fusion semantic prompts as described above.

[0015] The beneficial effects of this invention are as follows: 1. Achieve deep semantic fusion of medical multimodal data to eliminate cross-modal semantic gaps. This solution encodes image data and text descriptions simultaneously, and completes the precise alignment of visual features and text semantics through an adaptive multi-layer fusion network. This breaks down information barriers between modalities and generates a unified feature embedding that combines visual recognizability with medical semantic relevance, providing high-quality feature support for subsequent segmentation tasks.

[0016] 2. Constructing a contrastive learning constraint for difficult-to-distinguish negative samples to improve model discrimination accuracy and anti-interference ability. This scheme uses a large language model to generate highly similar, difficult-to-distinguish negative samples, and combines this with a negative sample contrastive loss function to force the model to accurately distinguish subtle medical semantic differences in lesion categories and anatomical locations. This effectively solves the problems of difficulty in fine-grained semantic discrimination of medical images and susceptibility to interference from similar samples, thus optimizing feature learning performance.

[0017] 3. Achieve accurate SAM segmentation driven by medical semantic cues, overcoming the semantic deficiencies of traditional geometric cues. This solution uses multimodal cues rich in medical semantics to replace traditional geometric cues in guiding SAM segmentation, fully integrating visual features with clinical text semantic information. This effectively solves problems such as segmentation bias and blurred boundaries, significantly improving the segmentation accuracy of lesion areas and anatomical structures. Attached Figure Description

[0018] Figure 1 This is a flowchart of the medical image segmentation method based on SAM multimodal and multi-stage fusion semantic prompting proposed in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the medical image segmentation system based on SAM multimodal and multi-stage fusion semantic prompting proposed in the second embodiment of the present invention; Figure 3 This is the overall logic flowchart of the present invention; The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0019] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0020] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] Please see Figure 1 The diagram shows a flowchart of a medical image segmentation method based on SAM multimodal multi-stage fusion semantic prompting proposed in the first embodiment of the present invention. This medical image segmentation method based on SAM multimodal multi-stage fusion semantic prompting includes steps S01 to S03, wherein: Step S01: Acquire multimodal target medical image data and perform feature extraction and multimodal feature fusion to obtain unified semantic feature embedding; It should be noted that in this embodiment, the feature extraction includes image feature extraction and text feature extraction, the multimodal feature fusion is based on self-attention modeling and bidirectional cross-attention, and the unified semantic feature embedding is based on multi-stage feature encoding and fusion processing. The multimodal target medical image data is used as the input image and divided into multiple fixed-size patch blocks. The initial visual features are obtained through linear mapping. Initial visual features are input into an image feature extraction network, which is based on four hierarchical stages. In each stage, features are modeled through multiple Swing Transformer blocks. Each Swing Transformer block alternately employs different multi-head attention mechanisms, including window multi-head self-attention and shifted window multi-head self-attention. The specific algorithm of the Swing Transformer block is as follows: , in, This indicates the output characteristics of the Swing Transformer block. This indicates a multi-head attention mechanism. This indicates the output characteristics of the previous Swing Transformer block. l Presentation layer index; At the end of each stage, feature downsampling is performed based on patch block stitching to obtain image features for the four stages; Text feature extraction is performed based on a staged encoder, which is based on the PubMedBERT encoder. The 12 Transformer layers in the staged encoder are divided into four stages: layers 1 to 6 are stage 1, layers 7 to 8 are stage 2, layers 9 to 10 are stage 3, and layers 11 to 12 are stage 4, thus obtaining text features for the four stages. Self-attention modeling is performed on the image features and text features at each stage to obtain self-attention-enhanced image features and self-attention-enhanced text features. Then, cross-modal information interaction is performed according to a bidirectional cross-attention mechanism. The cross-modal information interaction includes vision-to-text fusion and text-to-vision fusion. The specific algorithm for the cross-modal information interaction is as follows: , , in, This represents the visual-to-text fusion feature. This represents the text-to-visual fusion feature. This indicates self-attention enhanced text features. This represents self-attention enhanced image features. and These represent text multilayer perceptron and image multilayer perceptron, respectively. This represents a bidirectional cross-attention mechanism; Visual-to-text fusion features and text-to-visual fusion features are used as inputs to the next stage of text encoding and image encoding to obtain a unified semantic feature embedding for the four-stage encoding fusion.

[0023] Step S02: Construct indistinguishable negative samples and perform differential feature loss optimization; It should be noted that in this embodiment, the difficult-to-distinguish negative samples are generated based on a large language model, and the differential feature loss optimization processing is based on dynamic weights and regularization constraints. Multiple difficult-to-distinguish negative samples are generated for each training sample based on the large language model. The difficult-to-distinguish negative samples are semantically similar to the training samples but differ in key medical meanings. Text samples that are difficult to distinguish from negative samples are encoded and embedded to obtain their features. Positive sample text samples are also encoded and embedded to obtain their features. The similarity between the image features and the features of both positive and negative samples is calculated. The specific algorithm for this similarity calculation is as follows: , , in, and These represent the similarity between image features and positive sample text features, and between negative sample text features that are difficult to distinguish, respectively. This represents the features of the transposed image. This represents the features of positive sample text. This indicates that it is difficult to distinguish negative sample text features. Indicates the temperature coefficient; A contrastive learning loss function for hard-to-distinguish negative samples is constructed and optimized using differential feature loss. The specific algorithm for the contrastive learning loss function for hard-to-distinguish negative samples is as follows: , , in, This indicates a contrastive learning loss that makes it difficult to distinguish between negative samples. Indicates the number of positive samples. Indicates the ordinal number of the positive sample. This represents the regularization coefficient for positive samples, and K represents the number of negative samples that are difficult to distinguish. This indicates that the ordinal numbers of negative samples are difficult to distinguish. This indicates that it is difficult to distinguish the weights of negative samples. Representing image features and the first The similarity of negative sample text features is difficult to distinguish. This represents the weight hyperparameter.

[0024] Step S03: Perform medical image segmentation based on unified semantic feature embedding to obtain the final segmentation mask; It should be noted that in this embodiment, the medical image segmentation is based on a lightweight adapter, an image encoder, a mask decoder, and a low-rank adaptation module. Unified semantic features are embedded into the input SAM model, and semantic cues are obtained based on the lightweight adapter. The lightweight adapter is based on a two-layer multilayer perceptron with a GELU activation function. The specific algorithm for the semantic cues is as follows: , in, Indicates semantic hints, Indicates a lightweight adapter. Represents unified semantic feature embedding; The image encoder obtains the image encoding, and then the image encoding and semantic prompts are input into the mask decoder. Based on the cross-modal attention mechanism, the image encoding is guided by the semantic prompts to obtain the final segmentation mask.

[0025] Based on the method in this embodiment, please refer to the overall logical flow of this method. Figure 3 Systematic experiments were conducted on four medical image datasets: Kvasir, BUSI, ISIC, and QATA-COVID, and compared with traditional segmentation models U-Net, U-Net++, and other SAM-based methods. To comprehensively evaluate the segmentation performance of each method, DiceSimilarity Coefficient (DSC) and Normalized Surface Dice (NSD) were selected as the main evaluation metrics, and their mean and variance were statistically analyzed. The results are shown in Tables 1 and 2.

[0026] Table 1. DSC index of segmentation results Table 1 shows that the method of this invention outperforms other methods in all four datasets, especially on the Kvasir dataset, where it achieves a 23.5% improvement over the traditional U-Net method, and on the QATA-Covid dataset, it achieves a 26.2% improvement over the MedSAM method.

[0027] Table 2. NSD Index of Segmentation Results Table 2 shows that the method of this invention outperforms other methods in four datasets, especially on the Kvasir dataset, where it improves performance by 22.6% compared to the traditional U-Net method, and on the QATA-Covid dataset, it improves performance by 28.5% compared to the MedSAM method.

[0028] Please see Figure 2The diagram shows a schematic of a medical image segmentation system based on SAM multimodal and multi-stage semantic fusion proposed in the second embodiment of the present invention. The system includes: Feature extraction module 10 is used to acquire multimodal target medical image data and perform feature extraction and multimodal feature fusion to obtain unified semantic feature embedding. The feature extraction includes image feature extraction and text feature extraction. The multimodal feature fusion is based on self-attention modeling and bidirectional cross-attention. The unified semantic feature embedding is based on multi-stage feature encoding and fusion processing. The negative sample loss optimization module 20 is used to construct hard-to-distinguish negative samples and perform differential feature loss optimization processing. The hard-to-distinguish negative samples are generated based on a large language model, and the differential feature loss optimization processing is based on dynamic weights and regularization constraints. The segmentation module 30 is used to segment medical images based on unified semantic feature embedding to obtain a final segmentation mask. The medical image segmentation is based on a lightweight adapter, an image encoder, a mask decoder, and a low-rank adaptation module.

[0029] The present invention also proposes a computer storage medium storing one or more programs that, when executed by a processor, implement the above-described medical image segmentation method based on SAM multimodal multi-stage fusion semantic prompts.

[0030] The present invention also proposes a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to realize the above-mentioned medical image segmentation method based on SAM multimodal multi-stage fusion semantic prompts.

[0031] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0032] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0033] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0034] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0035] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A medical image segmentation method based on SAM (Multimodal Multi-stage Fusion Semantic Prompt), characterized in that, include: Multimodal target medical image data is acquired and feature extraction and multimodal feature fusion are performed to obtain unified semantic feature embedding. The feature extraction includes image feature extraction and text feature extraction. The multimodal feature fusion is based on self-attention modeling and bidirectional cross-attention. The unified semantic feature embedding is based on multi-stage feature encoding and fusion processing. Difficult-to-distinguish negative samples are constructed and differential feature loss is optimized. The difficult-to-distinguish negative samples are generated based on a large language model, and the differential feature loss optimization is based on dynamic weights and regularization constraints. Medical image segmentation is performed based on unified semantic feature embedding to obtain a final segmentation mask. The medical image segmentation is based on a lightweight adapter, an image encoder, a mask decoder, and a low-rank adaptation module.

2. The medical image segmentation method based on SAM with multimodal and multi-stage semantic fusion according to claim 1, characterized in that, The step of acquiring multimodal target medical image data and performing feature extraction and multimodal feature fusion to obtain unified semantic feature embedding specifically includes: Multimodal target medical image data is used as input images and divided into multiple fixed-size patch blocks. Initial visual features are obtained through linear mapping. Initial visual features are input into an image feature extraction network, which is based on four hierarchical stages. In each stage, features are modeled through multiple Swing Transformer blocks. Each Swing Transformer block alternately employs different multi-head attention mechanisms, including window multi-head self-attention and shifted window multi-head self-attention. The specific algorithm of the Swing Transformer block is as follows: ， in, This indicates the output characteristics of the Swing Transformer block. This indicates a multi-head attention mechanism. This indicates the output characteristics of the previous Swing Transformer block. l Presentation layer index; At the end of each stage, feature downsampling is performed based on patch block stitching to obtain image features for the four stages; Text feature extraction is performed based on a staged encoder, which is based on the PubMedBERT encoder. The 12 Transformer layers in the staged encoder are divided into four stages: layers 1 to 6 are stage 1, layers 7 to 8 are stage 2, layers 9 to 10 are stage 3, and layers 11 to 12 are stage 4, thus obtaining text features for the four stages. Self-attention modeling is performed on the image features and text features at each stage to obtain self-attention-enhanced image features and self-attention-enhanced text features. Then, cross-modal information interaction is performed according to a bidirectional cross-attention mechanism. The cross-modal information interaction includes vision-to-text fusion and text-to-vision fusion. The specific algorithm for the cross-modal information interaction is as follows: ， ， in, This represents the visual-to-text fusion feature. This represents the text-to-visual fusion feature. This indicates self-attention enhanced text features. This represents self-attention enhanced image features. and These represent text multilayer perceptron and image multilayer perceptron, respectively. This represents a bidirectional cross-attention mechanism; Visual-to-text fusion features and text-to-visual fusion features are used as inputs to the next stage of text encoding and image encoding to obtain a unified semantic feature embedding for the four-stage encoding fusion.

3. The medical image segmentation method based on SAM with multimodal and multi-stage semantic fusion according to claim 1, characterized in that, The steps of constructing indistinguishable negative samples and performing differential feature loss optimization specifically include: Based on a large language model, multiple indistinguishable negative samples are generated for each training sample. The indistinguishable negative samples are semantically similar to the training samples but differ in key medical meanings. Text samples that are difficult to distinguish from negative samples are encoded and embedded to obtain their features. Positive sample text samples are also encoded and embedded to obtain their features. The similarity between the image features and the features of both positive and negative samples is calculated. The specific algorithm for this similarity calculation is as follows: ， ， in, and These represent the similarity between image features and positive sample text features, and between negative sample text features that are difficult to distinguish, respectively. This represents the features of the transposed image. This represents the features of positive sample text. This indicates that it is difficult to distinguish negative sample text features. Indicates the temperature coefficient; A contrastive learning loss function for hard-to-distinguish negative samples is constructed and optimized using differential feature loss. The specific algorithm for the contrastive learning loss function for hard-to-distinguish negative samples is as follows: ， ， in, This indicates a contrastive learning loss that makes it difficult to distinguish between negative samples. Indicates the number of positive samples. Indicates the ordinal number of the positive sample. This represents the regularization coefficient for positive samples, and K represents the number of negative samples that are difficult to distinguish. This indicates that the ordinal numbers of negative samples are difficult to distinguish. This indicates that it is difficult to distinguish the weights of negative samples. Representing image features and the first The similarity of negative sample text features is difficult to distinguish. This represents the weight hyperparameter.

4. The medical image segmentation method based on SAM with multimodal and multi-stage semantic fusion according to claim 1, characterized in that, The step of performing medical image segmentation based on unified semantic feature embedding to obtain the final segmentation mask specifically includes: Unified semantic features are embedded into the input SAM model, and semantic cues are obtained based on a lightweight adapter. The lightweight adapter is based on a two-layer multilayer perceptron with a GELU activation function. The specific algorithm for the semantic cues is as follows: ， in, Indicates semantic hints, Indicates a lightweight adapter. Represents unified semantic feature embedding; The image encoder obtains the image encoding, and then the image encoding and semantic prompts are input into the mask decoder. Based on the cross-modal attention mechanism, the image encoding is guided by the semantic prompts to obtain the final segmentation mask.

5. A medical image segmentation system based on SAM (Multimodal Multi-stage Fusion Semantic Prompt), characterized in that, include: The feature extraction module is used to acquire multimodal target medical image data and perform feature extraction and multimodal feature fusion to obtain unified semantic feature embedding. The feature extraction includes image feature extraction and text feature extraction. The multimodal feature fusion is based on self-attention modeling and bidirectional cross-attention. The unified semantic feature embedding is based on multi-stage feature encoding and fusion processing. The negative sample loss optimization module is used to construct hard-to-distinguish negative samples and perform differential feature loss optimization processing. The hard-to-distinguish negative samples are generated based on a large language model, and the differential feature loss optimization processing is based on dynamic weights and regularization constraints. The segmentation module is used to segment medical images based on unified semantic feature embedding to obtain a final segmentation mask. The medical image segmentation is based on a lightweight adapter, an image encoder, a mask decoder, and a low-rank adaptation module.

6. A storage medium, characterized in that, The storage medium stores one or more programs that, when executed by a processor, implement the medical image segmentation method based on SAM multimodal multi-stage fusion semantic prompting as described in any one of claims 1-4.

7. A computer device, characterized in that, The computer device includes a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the medical image segmentation method based on SAM with multimodal and multi-stage fusion semantic prompts as described in any one of claims 1-4.

Images