Multimodal zero-shot medical image segmentation method based on visual-linguistic cue augmentation

Abstract

The application is a multimodal zero-shot medical image segmentation method based on visual-linguistic prompt enhancement. In the basic semantic generation and prompt semantic enrichment stage, the 'from coarse to fine' prompt generation mechanism uses VQA to dynamically extract the fine-grained attributes (such as shape and color) specific to the current target in the reasoning stage, to construct accurate initial prompts, and extracts factual attributes + GPT extended semantic library through BLIP. In the prompt sorting and screening stage, the cross-modal prompt sorting and screening strategy uses GLIP to calculate the cosine similarity between the text prompt and the potential target region features of the image, and eliminates the illusion of LLM. In the final visual prompt generation and segmentation stage, the descending sequence is spliced to guide. By combining visual-linguistic pre-training models and large language models, the limitations of single-modal information are broken, and high-precision zero-shot segmentation can be achieved without any medical image annotation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, artificial intelligence and medical image analysis technology, specifically to a multimodal zero-shot medical image segmentation method and system based on visual-language cues enhancement, which utilizes visual-language pre-trained models (VLPMs), large language models (LLMs) and segmentation base models (SAM) to achieve accurate zero-shot segmentation of medical images without manual annotation. Background Technology

[0002] Medical image segmentation is a crucial step in computer-aided diagnosis and clinical treatment planning, aiming to accurately extract lesions or organ regions from complex medical images. Current mainstream deep learning-based segmentation methods heavily rely on large amounts of pixel-level finely annotated medical image data. However, acquiring high-quality medical annotation data is extremely costly and involves data privacy protection issues, resulting in severely insufficient generalization ability of fully supervised models when faced with unseen categories or new modalities.

[0003] In recent years, segmentation models such as SegmentAnythingModel (SAM) have demonstrated remarkable zero-shot generalization capabilities in natural scenes. However, due to the significant "domain shift" between medical images and natural images in terms of imaging physics, grayscale distribution, and texture features, directly applying SAM to medical images often fails to accurately identify lesions. To address this issue, the closest existing techniques (such as SaLIP or TV-SAM) attempt to combine vision-language pre-trained models (such as GLIP), large language models (such as GPT), and SAM, utilizing textual cues to drive the model to generate bounding boxes, thereby guiding SAM for segmentation.

[0004] However, combining these three models in the medical field faces significant challenges, and existing methods of combination have the following major drawbacks: First, medical cues have limited semantic depth and poor feature activation capabilities. Existing technologies only use single medical category terms as cues. These simple terms lack descriptions of fine-grained attributes such as lesion shape, color, and texture, making it difficult to fully activate the latent medical knowledge in a multimodal large model, resulting in extremely inaccurate initial target localization.

[0005] Second, large language models suffer from "illusion," introducing significant localization noise. To enrich semantics, some existing technologies (such as TV-SAM) attempt to have large language models (LLMs) directly generate medical descriptions. However, because LLMs lack visual perception of the current input image, they are prone to generating "illusion" text that does not match the image facts (such as describing "blurred boundaries" as "clear boundaries"). Existing technologies lack cross-modal verification and filtering mechanisms; directly using these noisy cues for visual localization will cause the generated bounding boxes to deviate completely from the lesion target.

[0006] Third, the lack of temporal logic in multi-prompt input interferes with the model's attention mechanism. Existing techniques typically employ random input or disordered stacking after receiving multiple text prompts. However, in vision-language localization models, the order of text input directly affects the weight allocation of cross-attention. Low-quality or irrelevant prompts appearing randomly at the beginning of the sequence can severely disrupt the visual model's attention mechanism, leading to redundant, overlapping, or even erroneous bounding boxes in the output, ultimately causing the final SAM segmentation boundary to collapse.

[0007] In summary, existing technologies for zero-shot segmentation of medical images still fail to address the critical issue of "how to automatically acquire fine-grained semantic cues that closely match the image and effectively filter out noise." Therefore, there is an urgent need for a medical image segmentation method capable of automatically enriching semantic cues and ranking and filtering them based on cue quality. Summary of the Invention

[0008] The purpose of this invention is to address the problems existing in the prior art by proposing a multimodal zero-shot medical image segmentation method and system based on visual-language cues enhancement (PE-SAM). This invention overcomes the limitations of single-modal information by combining a visual-language pre-trained model and a large language model, achieving high-precision zero-shot segmentation without any medical image annotation.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a multimodal zero-shot medical image segmentation method based on visual-linguistic cue enhancement, the segmentation method comprising the following: Basic semantic generation: Acquire the input medical image, use a vision-language pre-trained model to perform preliminary detection and visual question answering on the medical image, extract the basic visual attributes for the current target medical condition, and construct a basic semantic set; Rich semantic prompts: The basic semantic set is input into a large language model for semantic diversity expansion and granular refinement, generating a set of candidate text prompts containing multiple fine-grained descriptions; Suggestion sorting and filtering: Each text suggestion in the candidate text suggestion set and the medical image are input into the visual-language localization model. The similarity score between the text suggestion and the potential target region features of the image is calculated. The text suggestions are sorted in descending order according to the similarity score, and the top N high-quality text suggestions are filtered out. Visual cue generation and segmentation: The top N high-quality text cuees are concatenated into a long text cue sequence according to the sorting result. The long text cue sequence and the corresponding medical image are input into the visual-language localization model to generate a fine visual bounding box. The medical image and the fine visual bounding box are input into the segmentation base model to generate and output the final segmentation mask, thus completing the segmentation of the medical image.

[0010] Furthermore, the vision-language pre-trained model consists of a vision-language localization model and a vision question answering and generation model; The specific processing method for generating the basic semantics is as follows: The original medical image and basic noun cues of the target medical category are input into the visual-language localization model. Using the nouns of the target medical category as cues, the visual-language localization model performs coarse localization on the original medical image to obtain a coarse bounding box. Based on the coarse bounding box or after expanding the coarse bounding box, a local image is cropped from the original medical image. The local images are input into a visual question answering and generation model. By setting a standardized visual question answering template, the basic visual attributes of each cropped local image are extracted. The basic visual attributes are combined with the basic noun prompts of the target medical category to form a basic semantic set.

[0011] Furthermore, the specific processing method for the rich semantics of the prompt is as follows: The basic semantic set is input into the large language model as known conditions, guiding the large language model to use its natural language generation capabilities from its medical and linguistic knowledge base to replace basic visual attributes with synonyms and add adverbs or specific morphological descriptions of different degrees for modification. Multiple descriptive phrases containing "shape + color + category noun" are generated to form a rich set of candidate text prompts.

[0012] Furthermore, the calculation process for the suggested sorting and filtering includes: Each text prompt in the candidate text prompt set and the original image are simultaneously input into the visual-language localization model. The text prompts generate text feature vectors through the visual-language localization model, and the original medical image generates regional visual feature vectors of multiple candidate regions through the visual-language localization model. Calculate the cosine similarity between the text feature vector and the regional visual feature vectors of multiple candidate regions. Then, calculate the average similarity score between the cosine similarity of a single text prompt and the regional visual feature vectors of all candidate regions. Sort the prompts in descending order according to this score and select the top N high-quality text prompts to form the final set of high-quality prompts.

[0013] Furthermore, the specific processing method for generating and segmenting the visual cues is as follows: The top N high-quality text prompts are concatenated into a long text prompt sequence in descending order. The long text prompt sequence and the original medical image are input into the visual-language localization model to predict a set of fine bounding boxes. The nonmaximum suppression algorithm is applied to remove redundant and overlapping predictions, resulting in a refined set of high-quality visual bounding boxes. The original medical image is input into the image encoder of the segmentation base model to extract high-dimensional image feature embeddings; a high-quality visual bounding box is used as a spatial prior and input into the cue encoder of the segmentation base model to be transformed into a cue embedding; the mask decoder of the segmentation base model receives the above image embeddings and cue embeddings, performs feature fusion through a bidirectional cross-attention mechanism, and finally outputs an accurate pixel-level target segmentation mask.

[0014] Secondly, the present invention provides a multimodal zero-shot medical image segmentation system based on visual-linguistic cue enhancement. The system executes the segmentation method described above, including: a basic semantic generation module, a cue semantic enrichment module, a cue sorting and filtering module, and a visual cue generation and segmentation module.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned multimodal zero-shot medical image segmentation method based on visual-language cues enhancement.

[0016] Fourthly, the present invention provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the aforementioned multimodal zero-shot medical image segmentation method based on visual-language cue enhancement.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention employs a multimodal collaborative approach, specifically manifested in the following ways: In the basic semantic generation and semantic enrichment stages, a "coarse-to-fine" prompt generation mechanism utilizes VQA to dynamically extract fine-grained attributes (such as shape and color) unique to the current target during the inference stage, used to construct accurate initial prompts. Factual attributes are extracted via BLIP, and the semantic library is expanded using GPT. In the prompt ranking and filtering stage, a cross-modal prompt ranking and filtering strategy is employed, utilizing GLIP to calculate the cosine similarity between text prompts and the features of potential target regions in the image, eliminating the illusion of LLM. In the final visual prompt generation and segmentation stage, a descending sequence concatenation guides the process. A text concatenation method that prioritizes high-quality prompts provides positive attention guidance for GLIP to generate high-quality visual priors. Specifically: First, it achieves completely zero-shot and manual-annotation-free segmentation. This invention constructs a collaborative framework integrating a large language model, a vision-language model, and a segmentation base model. The entire process is automated, avoiding subjective biases caused by manually designed prompts. It breaks through the bottleneck of traditional medical image segmentation, which highly depends on pixel-level labeled data. For zero-shot segmentation, no training is required, significantly reducing application costs.

[0018] Secondly, it effectively bridges the "domain shift" gap between medical semantics and visual features. Through a "coarse-to-fine" semantic enrichment strategy, it transforms single, abstract medical category terms into fine-grained descriptions of basic visual attributes, including shape, color, and texture. This rich semantic information compensates for the lack of medical expertise in general-purpose models, enabling them to focus more precisely on lesion areas.

[0019] Furthermore, the noise resistance and localization accuracy of the model are significantly improved. This invention innovatively introduces a prompt ranking and filtering mechanism based on the visual-language localization model GLIP, accurately eliminating "illusions" and irrelevant descriptions generated by large language models through cosine similarity in the feature space. Building upon this, a sequence input strategy of "arranging and concatenating in descending order of quality" is adopted, providing the visual-language localization model with a strong "positive feedback" initial context, avoiding noise interference from low-quality prompts, thereby generating high-quality visual bounding boxes comparable to those annotated manually.

[0020] Finally, it exhibits extremely strong cross-modal generalization robustness. On publicly available multimodal medical datasets (including ultrasound, endoscopy, X-ray, pathological slides, etc.), this invention can still output highly compliant segmentation boundaries even when lesion boundaries are blurred and contrast is extremely low. It not only surpasses existing unsupervised segmentation methods, but its performance curve also significantly approaches the upper limit of fully supervised theory, providing a highly promising general segmentation solution for complex real-world clinical scenarios.

[0021] Taking the Breast Ultrasound (BUSI) dataset as an example, when existing baseline methods (such as GSAM) use only a single medical term (such as "tumor") as a cue, they are easily affected by background tissue interference on ultrasound images with blurred boundaries, resulting in a Dice similarity coefficient of only 0.731 on the BUSI dataset. However, by introducing the PE-SAM framework of this invention, rich attributes such as "irregular shape" and "low echo" are extracted and subjected to precise cross-modal ranking and filtering, providing the model with extremely high-quality visual bounding box priors. Ultimately, the Dice coefficient of this invention on the BUSI dataset is improved to 0.782. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the medical image segmentation method based on visual-language cues enhancement provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram illustrating a process with rich semantic prompts.

[0024] Figure 3 This is a diagram illustrating the sorting and filtering process.

[0025] Figure 4 This is an example diagram illustrating the actual usage effect of PE-SAM provided in this embodiment of the invention. Detailed Implementation

[0026] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of protection of this application.

[0027] This invention proposes a multimodal zero-shot medical image segmentation method based on visual-language prompt enhancement (PE-SAM, PromptEnrichmentforSegmentAnythingModel). Unlike traditional methods that involve tedious fine-tuning of the segmentation model, the core idea of ​​this invention is a "coarse-to-fine" cross-modal knowledge-guided approach: using a visual-language pre-trained model to generate basic descriptions, using a large language model to enrich semantics, removing noise through similarity ranking, and finally converting high-quality text into accurate bounding boxes, thereby driving the SAM model to complete the segmentation.

[0028] Example 1 The multimodal zero-shot medical image segmentation method based on visual-language cues enhancement in this embodiment specifically includes the following four core steps: Step 1: Basic Semantic Generation The problem to be solved is how to establish a preliminary connection between visual signals and linguistic symbols to prevent large models from producing "illusions" that are detached from the facts of the images.

[0029] Coarse localization based on category prior: Let the input original medical image be... The basic noun prompts for the target medical category are: (e.g., "tumor", "lung"). Preliminary detection is performed using the pre-trained visual-language localization model GLIP, generating coarse bounding boxes. .

[0030] Based on the rough bounding box The relative size of the original medical image is proportionally expanded to include more contextual information, resulting in a larger coarse bounding box, and the original medical image is then used as the basis for this expansion. The image is cropped using the expanded coarse bounding box to extract a local portion of the image. .

[0031] in The coarse bounding boxes represent the predicted output and are used to define potential lesion areas and filter the background. The GroundedLanguage-ImagePre-training (GL-Image) is represented as a mapping function, which here is the vision-language localization model. This represents the original medical image input. The basic noun indicating the target medical category is "tumor" or "lungs".

[0032] Attribute extraction based on visual question answering: extracting the cropped local image The input is fed into the unified visual question answering and generative model BLIP. Utilizing the visual question answering (VQA) capabilities of BLIP, basic visual attributes (such as the shape of the target) of each cropped image are extracted by inputting standardized questions (e.g., "What is the shape of this object?", "What is the color or texture of this object?"). and color / texture (The examples use shape and color / texture as the most preferred representations.) By combining the basic visual attributes extracted above with the basic noun cues of the target medical category, a basic semantic set is constructed:

[0033] in This represents the base text set. This refers to the answer for the target shape attribute extracted through visual question answering and generative models, such as "cone". This indicates the answer to the target color or texture attribute extracted by the visual question answering model, such as "dark". Basic noun prompts indicating the target medical category.

[0034] Step 2: Provide rich semantic prompts (see...) Figure 2 ) The problem it addresses is that single attribute descriptions cannot fully activate the model's feature responses and lack semantic form.

[0035] Basic semantic set Given these conditions, an instruction template is constructed and input into a large language model (such as GPT). The instruction template refers to: Input: I have a medical image where the target object is [Category:] After initial observation, the basic visual attribute of this object is that its shape is similar to... The color / texture is similar to .

[0036] Task: Based on the above basic visual attributes, use your medical and linguistic knowledge to generate 20 more detailed and diverse descriptive phrases.

[0037] Requirements: Replace basic attribute words with different synonyms, add degree adverbs or specific morphological descriptions, and output a simple list of noun phrases in the form of '[shape][color][noun]'.

[0038] Diversity expansion and granular refinement: The large language model GPT is guided to use its medical and linguistic knowledge base to perform synonym replacement on basic visual attributes (such as expanding "circle" to "ring", "spherical", etc.) and add adverbs of different degrees (such as slightly, very) or specific morphological descriptions.

[0039] The final result is a more detailed and diverse list of N descriptive phrases, forming a rich set of candidate text prompts:

[0040] in This represents the expanded, richer set of candidate text prompts (RichTextpromptset). This represents a generative large language model function (such as ChatGPT). The input represents the basic semantic set (serving as constraints for generation). L represents the context instruction / image context, constraining the large model to remain within the context of the current medical image. Represents the first to the second generated by the large language model. X Each sample in the candidate text prompt set corresponds to an original image and is a fine-grained descriptive phrase.

[0041] Step 3: Prompt for sorting and filtering (see...) Figure 3 ) The problem addressed is that the prompts generated by GPT may contain noise, which can lead to positioning errors if used directly.

[0042] Feature Encoding and Interaction: Encoding the Set of Candidate Text Hints Each text prompt and the original image Simultaneously input the visual-speech localization model GLIP. Text prompts. The text feature vector P is generated using GLIP, from the original medical image. GLIP is used to generate regional visual feature vectors R for multiple candidate regions (potential target regions in the image).

[0043] Region-Text Alignment Metric: Calculates the cosine similarity between the text feature vector and the visual feature vector of a given region. :

[0044] Since there are multiple candidate regions in the image, the average similarity score between the text prompt and the visual feature vectors of all candidate regions is calculated. .

[0045] Where, N box Represents all candidate regions. This represents the visual feature vector of the i-th candidate region.

[0046] Top-N filtering: Selecting the set of candidate text hints All text prompts are based on average similarity score. Sort in descending order Extract the first part after sorting. A set of high-quality text prompts constitutes the final set of excellent prompts. .

[0047] Step 4: Visual Cue Generation and Segmentation The problem to be solved is how to convert semantic information into high-precision pixel-level masks.

[0048] Visual cue generation based on sorted sequences: This involves selecting a set of high-quality cue items. The sentences are concatenated in descending order of score to form a long text prompt sequence:

[0049] in This represents a long text prompt sequence generated by concatenating and arranging the prompts in descending order.

[0050] This long text cue sequence, by placing high-quality text cues at the beginning, can provide strong positive attention guidance for the text encoder of the visual-language localization model.

[0051] Original medical images and long text prompt sequence The visual-speech localization model GLIP is input, predicting a set of fine-grained bounding boxes. Non-maximum suppression (NMS) is then applied to remove redundancy and overlap, resulting in a refined, high-quality set of visual bounding boxes.

[0052] in This indicates the final set of high-quality visual bounding boxes (FinalBoundingBoxes) that has been refined and output, which will serve as visual cues for subsequent use. This represents the Non-Maximum Suppression algorithm, used to eliminate overlapping redundant boxes. The GLIP model is based on a sequence of long text prompts input. The predicted set of fine-grained visual bounding boxes.

[0053] Zero-shot segmentation combined with SAM: transforming raw medical images The image encoder inputting SAM extracts high-dimensional image features and embeds them; high-quality visual bounding boxes are then used. The cue encoder, which serves as the spatial prior input to the SAM, transforms it into a cue embedding. The SAM mask decoder receives the aforementioned image embedding and cue embedding, performs feature fusion through a bidirectional cross-attention mechanism, and finally outputs a precise pixel-level target segmentation mask.

[0054] in Represents a precise pixel-level target segmentation mask (MaskPrediction). This represents the mapping function of the SegmentAnythingModel. This represents the original input medical image. This represents a high-quality set of visual bounding boxes, which serves as a spatial prior visual cues (Visual Prompt) in SAM.

[0055] In this invention, the underlying mathematical operations for GLIP attribute extraction, GPT text generation, and SAM mask decoding can all be implemented based on existing technologies.

[0056] Example 2 In practical applications, this embodiment uses the PyTorch deep learning framework for system construction, including: The basic semantic generation module is configured to extract basic visual attributes of the input medical image using a vision-language pre-trained model. These basic visual attributes include shape, color, and texture, and are combined with the medical category name to generate a basic semantic set. The semantic enrichment module is configured to use a large language model to expand the basic semantic set in terms of diversity and granularity, and generate a candidate text prompt set. The prompt sorting and filtering module is configured to calculate the similarity score between candidate text prompts and the features of potential target regions in the image, and filter out the top N high-quality text prompts with the highest similarity scores. The visual cue generation and segmentation module is configured to concatenate the high-quality text cuees into a long text cue sequence according to the sorting result, input the long text cue sequence and the corresponding medical image into the visual-language localization model to generate a fine visual bounding box, and input the medical image and the fine visual bounding box into the segmentation base model to generate and output the final segmentation mask.

[0057] In this approach, the visual question answering and generation model BLIP of the basic semantic generation module adopts the ViT-B architecture and has been fine-tuned; the large language model uses the GPT interface (such as GPT-3.5 / 4); the visual-language localization model GLIP uses GLIP-L weights; and the segmentation base model uses a ViT-H variant of SAM. This method is a zero-shot inference architecture and does not involve retraining or fine-tuning the model weights. The datasets used (BUSI, Kvasir-SEG, etc.) are all publicly available validation sets directly used for inference testing.

[0058] Table 1 shows the comparison results of segmentation using the segmentation method of this invention with other existing methods on multiple datasets. The Dice coefficient is used to characterize the quality of the segmentation results, with a larger Dice coefficient indicating better segmentation.

[0059] Table 1

[0060] The datasets used are: X-ray: X-ray (chest X-ray) dataset / ChestXrayMasksandLabels dataset BUSI (Breast Ultrasound) Dataset Kvasir-SEG: A dataset for endoscopy of gastrointestinal polyps MoNuSeg: (Multi-organ cell nucleus microscopy) dataset ISIC18: (Dermoscopy for Skin Lesions) Dataset GlaS: (Colonial Gland Microscopy) Dataset Experiments on the publicly available breast ultrasound dataset (BUSI) and skin lesion dataset (ISIC) show that when the baseline model (GSAM) uses only a single medical terminology as a cue, it is easily affected by background tissue interference, leading to missed detections. However, after introducing the PE-SAM framework of this invention, the automatically generated rich cues such as "dark color, irregular boundary, and hypoechoic area" are accurately ordered, ensuring that the generated visual prompt closely fits the lesion edge. Ultimately, the Dice similarity coefficient between the segmentation mask (prediction) generated by SAM and the doctor's manual annotation (Ground Truth) is significantly improved, as shown in the results. Figure 4 As shown. Furthermore, ablation experiments have demonstrated that prioritizing descending order of similarity (highest quality first) for stitching improves positioning accuracy compared to random or ascending order stitching.

[0061] This invention uses cosine similarity for cross-modal alignment measurement. It calculates the cosine similarity between the plain text generated by LLM and the actual image region features extracted by GLIP, and uses visual features to "verify" the text features, thereby accurately removing hallucination noise. This avoids the drawback of large language models in multimodal medical segmentation being prone to "hallucinations" and generates descriptions that are more consistent with the actual content of the current medical image.

[0062] The above description is merely a preferred embodiment of the present invention. Based on the above description, those skilled in the art can make various changes and modifications without departing from the technical concept of this invention. The technical scope of this invention is not limited to the contents of the specification but must be determined according to the scope of the claims.

[0063] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A multimodal zero-shot medical image segmentation method based on visual-linguistic cue enhancement, the segmentation method comprising the following: Basic semantic generation: Acquire the input medical image, use a vision-language pre-trained model to perform preliminary detection and visual question answering on the medical image, extract the basic visual attributes for the current target medical condition, and construct a basic semantic set; Rich semantic prompts: The basic semantic set is input into a large language model for semantic diversity expansion and granular refinement, generating a set of candidate text prompts containing multiple fine-grained descriptions; Suggestion sorting and filtering: Each text suggestion in the candidate text suggestion set and the medical image are input into the visual-language localization model. The similarity score between the text suggestion and the potential target region features of the image is calculated. The text suggestions are sorted in descending order according to the similarity score, and the top N high-quality text suggestions are filtered out. Visual cue generation and segmentation: The top N high-quality text cuees are concatenated into a long text cue sequence according to the sorting result. The long text cue sequence and the corresponding medical image are input into the vision-language localization model to generate a fine visual bounding box. The medical image and the fine visual bounding box are input into the segmentation base model to generate and output the final segmentation mask, thus completing the segmentation of the medical image.

2. The method according to claim 1, characterized in that, The vision-language pre-trained model consists of a vision-language localization model and a vision question answering and generation model. The specific processing method for generating the basic semantics is as follows: The original medical image and basic noun cues of the target medical category are input into the visual-language localization model. Using the nouns of the target medical category as cues, the visual-language localization model performs coarse localization on the original medical image to obtain a coarse bounding box. Based on the coarse bounding box or after expanding the coarse bounding box, a local image is cropped from the original medical image. The local images are input into a visual question answering and generation model. By setting a standardized visual question answering template, the basic visual attributes of each cropped local image are extracted. The basic visual attributes are combined with the basic noun prompts of the target medical category to form a basic semantic set.

3. The method according to claim 1, characterized in that, The specific processing method for the rich semantics of the prompt is as follows: The basic semantic set is input into the large language model as known conditions, guiding the large language model to use its natural language generation capabilities from its medical and linguistic knowledge base to replace basic visual attributes with synonyms and add adverbs or specific morphological descriptions of different degrees for modification. Generate multiple descriptive phrases containing "shape + color + category noun" to form a rich set of candidate text prompts.

4. The method according to claim 1, characterized in that, The calculation process for the sorting and filtering of the prompts includes: Each text prompt in the candidate text prompt set and the original image are simultaneously input into the visual-language localization model. The text prompts generate text feature vectors through the visual-language localization model, and the original medical image generates regional visual feature vectors of multiple candidate regions through the visual-language localization model. Calculate the cosine similarity between the text feature vector and the regional visual feature vectors of multiple candidate regions. Then, calculate the average similarity score between the cosine similarity of a single text prompt and the regional visual feature vectors of all candidate regions. Sort the prompts in descending order according to this score and select the top N high-quality text prompts to form the final set of high-quality prompts.

5. The method according to claim 1, characterized in that, The specific processing method for generating and segmenting the visual cues is as follows: The top N high-quality text prompts are concatenated into a long text prompt sequence in descending order. The long text prompt sequence and the original medical image are input into the visual-language localization model to predict a set of fine bounding boxes. The nonmaximum suppression algorithm is applied to remove redundant and overlapping predictions, resulting in a refined set of high-quality visual bounding boxes. The original medical image is input into the image encoder of the segmentation base model to extract high-dimensional image feature embeddings; a high-quality visual bounding box is input as a spatial prior into the cue encoder of the segmentation base model and converted into cue embeddings. The mask decoder of the segmentation base model receives the above image embedding and cue embedding, performs feature fusion through a bidirectional cross-attention mechanism, and finally outputs an accurate pixel-level target segmentation mask.

6. A multimodal zero-shot medical image segmentation system based on visual-verbal cues, characterized in that, The system executes the segmentation method according to any one of claims 1-5, including: The basic semantic generation module is configured to extract basic visual attributes of the input medical image using a vision-language pre-trained model. These basic visual attributes include shape, color, and texture, and are combined with the medical category name to generate a basic semantic set. The semantic enrichment module is configured to use a large language model to expand the basic semantic set in terms of diversity and granularity, and generate a candidate text prompt set. The prompt sorting and filtering module is configured to calculate the similarity score between candidate text prompts and the features of potential target regions in the image, and filter out the top N high-quality text prompts with the highest similarity scores. The visual cue generation and segmentation module is configured to concatenate the high-quality text cuees into a long text cue sequence according to the sorting result, input the long text cue sequence and the corresponding medical image into the visual-language localization model to generate a fine visual bounding box, and input the medical image and the fine visual bounding box into the segmentation base model to generate and output the final segmentation mask.

7. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the multimodal zero-shot medical image segmentation method based on visual-language cues enhancement as described in any one of claims 1-5.

8. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multimodal zero-shot medical image segmentation method based on visual-language cue enhancement as described in any one of claims 1-5.

Images