Zero-shot referential image segmentation method based on text-aware adapter

By embedding local and global adapters in the CLIP image encoder, deep interaction between visual and linguistic modalities is achieved, solving the problem of feature misalignment in existing methods and improving the robustness and accuracy of zero-shot reference image segmentation.

CN121725007BActive Publication Date: 2026-04-21INSTITUTE OF MATERIALS & INTELLIGENT MANUFACTURING JIANGXI ACADEMY OF SCIENCES +2
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
INSTITUTE OF MATERIALS & INTELLIGENT MANUFACTURING JIANGXI ACADEMY OF SCIENCES
Filing Date
2026-02-25
Publication Date
2026-04-21

AI Technical Summary

Technical Problem

Existing CLIP-based referential image segmentation methods lack deep interaction between visual and linguistic modalities during the encoding stage, resulting in feature misalignment and poor segmentation robustness in low-quality or blurry image scenarios, often leading to blurred boundaries or target recognition errors.

Method used

By embedding local and global adapters into a frozen pre-trained dual-branch CLIP image encoder, deep interaction between visual and linguistic modalities is achieved. Local adapters are used for region text alignment, and global adapters are used for noise-resistant enhancement, generating fine-grained aligned local enhancement features and noise-resistant global enhancement features.

Benefits of technology

It effectively solves the modality misalignment problem in the feature extraction stage, improves the robustness and segmentation accuracy of the segmentation model in blurry or complex scenarios, and is significantly better than existing methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121725007B_ABST
    Figure CN121725007B_ABST
Patent Text Reader

Abstract

The application discloses a zero-shot referring image segmentation method based on a text-aware adapter; the method comprises the following steps: acquiring an original image and referring text, generating a candidate mask by using a pre-trained segmentation model, and constructing a local input image and a global input image; a double-branch CLIP image encoder is used to embed a training-free local adapter and a global adapter in a frozen parameter Transformer module; the local adapter is guided and gated by a SAM mask fusion mechanism, the local text is embedded by the local adapter, and the local text and regional visual features are fused to realize fine-grained region alignment; the global adapter uses variance-based feature enhancement and hyperbolic tangent robust similarity calculation to retain anti-interference global context information; the fused double-branch output features score the candidate mask, and finally output the segmentation result; the application does not need parameter fine-tuning, effectively solves the problem of multi-modal feature misplacement, and significantly improves the accuracy and robustness of zero-shot referring segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing, specifically to a zero-sample referential image segmentation method based on a text-aware adapter. Background Technology

[0002] Image segmentation of reference aims to segment specific targets in an image based on natural language representations. Although traditional fully supervised methods have acceptable accuracy, they rely on a large number of pixel-level annotations, which are costly and limit their promotion in large-scale applications. In recent years, with the development of large-scale vision-language models, zero-shot reference segmentation has become a research hotspot, aiming to achieve segmentation without training for specific categories.

[0003] However, existing CLIP-based pronoun image segmentation methods still have significant shortcomings. First, existing methods typically adopt an extraction-then-fusion paradigm, that is, using independent image encoders and text encoders to extract features separately, and then fusing them in the decoder. This architecture results in a lack of interaction between the visual and linguistic modalities during the encoding stage. The image encoder only acts as a passive feature extractor, and the generated features are often general and lack discriminativeness for specific pronoun texts, leading to feature misalignment. Second, although some methods introduce SAM to generate candidate masks, CLIP is often only used to classify the proposal boxes generated by SAM, lacking deep feature-level fusion. In addition, in low-quality or blurry image scenes, existing methods have difficulty effectively utilizing global contextual information, resulting in poor segmentation robustness and often causing problems such as blurred segmentation boundaries or target recognition errors.

[0004] Therefore, it is evident that how to achieve deep alignment between visual features and text semantics during the encoding stage without retraining or fine-tuning parameters, and how to transform passive feature extraction into active text perception, is a technical challenge that urgently needs to be addressed in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a zero-sample referential image segmentation method based on a text-aware adapter. Its core lies in: by embedding a training-free local adapter and a global adapter inside a frozen pre-trained dual-branch CLIP image encoder, the CLIP image encoder is transformed from a passive feature extractor into an active text-aware adapter, realizing deep and fine-grained interaction between visual and linguistic modalities during the encoding stage.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a zero-shot referential image segmentation method based on a text-aware adapter, comprising the following steps:

[0007] Step S1: Collect the original image to be segmented and the referential text, and use the pre-trained segmentation model to generate a candidate mask set for the original image. Based on the candidate mask set, construct the local input image and the global input image respectively.

[0008] Step S2: Perform multi-granularity feature extraction on the referential text, extracting global text embeddings at the whole sentence level and local text embeddings at the phrase level respectively;

[0009] Step S3: Construct a dual-branch CLIP image encoder containing a local perception branch and a global perception branch, keep the parameters of the dual-branch CLIP image encoder frozen, and embed the training-free local adapter and global adapter in the Transformer module of the dual-branch CLIP image encoder.

[0010] Step S4: In the local perception branch, the local input image is input into the CLIP image encoder for extraction to obtain regional visual features; the local text embedding is fused with the regional visual features using a local adapter, and fine-grained aligned local enhancement features are generated through weighted regional feature aggregation and gating fusion mechanism; In the global perception branch, the global input image is input into the CLIP image encoder for extraction to obtain global visual features; the global text embedding is fused with the global visual features using a global adapter, and noise-resistant global enhancement features are generated through variance-based feature enhancement and robust similarity calculation.

[0011] Step S5: In the inference stage, local enhancement features and global enhancement features are fused. The semantic similarity score between each candidate mask in the candidate mask set and the referential text is calculated using the fused features. The candidate mask with the highest score is selected as the final referential image segmentation result, thus completing the zero-shot referential image segmentation task.

[0012] Furthermore, the processing procedure for the local sensing branch is as follows:

[0013] Step S411: Spatial downsampling and flattening are performed on the candidate mask to obtain a candidate mask vector aligned with the visual features of the region;

[0014] Step S412: Based on the aligned candidate mask vectors, perform weighted aggregation of the region visual features, and extract the region visual features corresponding to each candidate mask. The calculation formula is as follows:

[0015] ;

[0016] in, Indicates the first The visual features of the regions corresponding to each candidate mask, where L represents the total number of image patches. Indicates the first Image blocks, Indicates the first The candidate mask in the th... Weights on each image patch Represents the first in the visual feature sequence Feature vectors of image patches Represents the numerical stability constant;

[0017] Step S413: Project the visual features of the regions corresponding to each candidate mask onto the text feature space, and calculate their semantic similarity with the local text embedding. The calculation formula is as follows:

[0018] ;

[0019] in, Indicates semantic similarity. Represents the projection matrix. This indicates a transpose operation for local text embedding. This means taking the larger of the two values. Indicates local text embedding;

[0020] Step S414: Generate a position weight matrix based on semantic similarity, and combine it with a gating fusion mechanism to embed local text into the visual feature sequence to obtain local enhanced features.

[0021] Furthermore, the specific implementation of the gating fusion mechanism in step S414 is as follows: gating weights are generated using the Sigmoid activation function, and local enhanced features are calculated according to the following formula:

[0022] ;

[0023] in, Indicates local enhancement features, Represents a sequence of visual features. Represents the fusion coefficient. Indicates the gating weight, This represents the position weight matrix based on semantic similarity. This represents the expanded text feature vector.

[0024] Furthermore, the processing procedure for the global perception branch is as follows:

[0025] Step S421: Calculate the weighted global features based on the channel variance of image patch features in the global visual features, and then perform weighted fusion with the CLS token features to obtain the enhanced global visual features. The calculation formula is as follows:

[0026] ;

[0027] in, This represents the enhanced global visual features. Represents the balance coefficient. Indicates the characteristics of the CLS token. This represents the natural exponential function. Indicates the first Channel variance of features of an image patch Let represent the temperature parameter, and k represent the index variable in the summation process. This iterates through all image patches, with values ​​ranging from 1 to L. Represents the feature vector of the k-th image patch Variance in the channel dimension;

[0028] Step S422: Project the enhanced global visual features onto the text feature space, and calculate its robust similarity with the global text embedding using the hyperbolic tangent function. The calculation formula is as follows:

[0029] ;

[0030] in, Indicates robust similarity, Represents the hyperbolic tangent function. This represents the transpose operation for global text embedding. Indicates global text embedding;

[0031] Step S423: Generate a global fusion weight matrix by combining robust similarity and spatial saliency maps;

[0032] Step S424: Inject global text embedding into global visual features based on the global fusion weight matrix to obtain global enhanced features. The calculation formula is as follows:

[0033] ;

[0034] in, Indicates global enhancement features, Represents the fusion coefficient. Represents the fusion weight matrix. Indicates a broadcast extension operation. This represents the transpose operation of the projection matrix.

[0035] Furthermore, the global fusion weight matrix in step S423 is generated as follows: for the location of the CLS token feature, the global fusion weight is the robust similarity; for the location of the image patch feature, the global fusion weight is the product of the robust similarity and the spatial saliency value of the corresponding location; the calculation formula is:

[0036] ;

[0037] in, Represents the position in the fusion weight matrix The weight value, Represents the index in the feature sequence, where Corresponding to the CLS token feature position, Corresponding image patch feature location, This represents the average saliency map extracted from all candidate masks.

[0038] Furthermore, in step S1, the local input image is constructed by multiplying the original image element-wise with each candidate mask to obtain the local input image corresponding to each candidate mask; the global input image is constructed by performing Gaussian blur processing on the areas of the original image not covered by the candidate masks to obtain the global input image with overall context preserved.

[0039] Furthermore, the multi-granularity feature extraction in step S2 includes: using CLIP text encoder to extract full sentence embeddings as global text embeddings; using natural language parsing tools to extract noun phrases in the referential text, and using CLIP text encoder to encode them as phrase embeddings as local text embeddings.

[0040] Furthermore, in step S3, both the local adapter and the global adapter are training-free modules, directly embedded into the Transformer module of the pre-trained dual-branch CLIP image encoder with frozen parameters.

[0041] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the instructions are executed by the processor, the processor implements the above-described zero-sample reference image segmentation method.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, can execute instructions to perform the above-described zero-sample reference image segmentation method.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This invention proposes a training-free adapter framework that transforms the CLIP encoder from a passive feature extractor into an active text perceptron by embedding local and global adapters within the frozen CLIP encoder, effectively solving the modality misalignment problem in the feature extraction stage.

[0045] The local adapter utilizes SAM prior knowledge to achieve accurate region-text alignment and enhance boundary details; the global adapter effectively suppresses background noise and improves the robustness of the segmentation model in blurry or complex scenes through variance enhancement and Tanh similarity mechanism.

[0046] Experimental results show that the zero-shot segmentation performance of the present invention is significantly better than the existing state-of-the-art methods on three benchmark datasets: RefCOCO, RefCOCO+, and RefCOCOg, verifying the effectiveness of the present invention in weakly supervised object detection tasks. Attached Figure Description

[0047] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0048] Referring to the following description and accompanying drawings, some specific embodiments of the present invention are disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention. However, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0049] like Figure 1 As shown, this embodiment proposes the following technical solution: a zero-shot referential image segmentation method based on a text-aware adapter, which includes the following steps:

[0050] Step S1: Collect the original image to be segmented and the referential text, and use the pre-trained segmentation model to generate a candidate mask set for the original image. Based on the candidate mask set, construct the local input image and the global input image respectively.

[0051] Step S2: Perform multi-granularity feature extraction on the referential text, extracting global text embeddings at the whole sentence level and local text embeddings at the phrase level respectively;

[0052] Step S3: Construct a dual-branch CLIP image encoder containing a local perception branch and a global perception branch, keep the parameters of the dual-branch CLIP image encoder frozen, and embed the training-free local adapter and global adapter in the Transformer module of the dual-branch CLIP image encoder.

[0053] Step S4: In the local perception branch, the local input image is input into the CLIP image encoder for extraction to obtain regional visual features; the local text embedding is fused with the regional visual features using a local adapter, and fine-grained aligned local enhancement features are generated through weighted regional feature aggregation and gating fusion mechanism; In the global perception branch, the global input image is input into the CLIP image encoder for extraction to obtain global visual features; the global text embedding is fused with the global visual features using a global adapter, and noise-resistant global enhancement features are generated through variance-based feature enhancement and robust similarity calculation.

[0054] Step S5: In the inference stage, local enhancement features and global enhancement features are fused. The semantic similarity score between each candidate mask in the candidate mask set and the referential text is calculated using the fused features. The candidate mask with the highest score is selected as the final referential image segmentation result, thus completing the zero-shot referential image segmentation task.

[0055] Furthermore, the processing procedure for the local sensing branch is as follows:

[0056] Step S411: Spatial downsampling and flattening are performed on the candidate mask to obtain a candidate mask vector aligned with the visual features of the region;

[0057] Step S412: Based on the aligned candidate mask vectors, perform weighted aggregation of the region visual features, and extract the region visual features corresponding to each candidate mask. The calculation formula is as follows:

[0058] ;

[0059] in, Indicates the first The visual features of the regions corresponding to each candidate mask, where L represents the total number of image patches. Indicates the first Image blocks, Indicates the first The candidate mask in the th... Weights on each image patch Represents the first in the visual feature sequence Feature vectors of image patches Represents the numerical stability constant;

[0060] Step S413: Project the visual features of the regions corresponding to each candidate mask onto the text feature space, and calculate their semantic similarity with the local text embedding. The calculation formula is as follows:

[0061] ;

[0062] in, Indicates semantic similarity. Represents the projection matrix. This indicates a transpose operation for local text embedding. This means taking the larger of the two values. Indicates local text embedding;

[0063] Step S414: Generate a position weight matrix based on semantic similarity, and combine it with a gating fusion mechanism to embed local text into the visual feature sequence to obtain local enhanced features.

[0064] Furthermore, the specific implementation of the gating fusion mechanism in step S414 is as follows: gating weights are generated using the Sigmoid activation function, and local enhanced features are calculated according to the following formula:

[0065] ;

[0066] in, Indicates local enhancement features, Represents a sequence of visual features. This represents the fusion coefficient, and its value is 0.2. Indicates the gating weight, This represents the position weight matrix based on semantic similarity. This represents the expanded text feature vector.

[0067] Furthermore, the processing procedure for the global perception branch is as follows:

[0068] Step S421: Calculate the weighted global features based on the channel variance of image patch features in the global visual features, and then perform weighted fusion with the CLS token features to obtain the enhanced global visual features. The calculation formula is as follows:

[0069] ;

[0070] in, This represents the enhanced global visual features. This represents the balance coefficient, and its value is 0.4. Indicates the characteristics of the CLS token. This represents the natural exponential function. Indicates the first Channel variance of features of an image patch Let represent the temperature parameter, and k represent the index variable in the summation process. This iterates through all image patches, with values ​​ranging from 1 to L. Represents the feature vector of the k-th image patch Variance in the channel dimension;

[0071] Step S422: Project the enhanced global visual features onto the text feature space, and calculate its robust similarity with the global text embedding using the hyperbolic tangent function. The calculation formula is as follows:

[0072] ;

[0073] in, Indicates robust similarity, Represents the hyperbolic tangent function. This represents the transpose operation for global text embedding. Indicates global text embedding;

[0074] Step S423: Generate a global fusion weight matrix by combining robust similarity and spatial saliency maps;

[0075] Step S424: Inject global text embedding into global visual features based on the global fusion weight matrix to obtain global enhanced features. The calculation formula is as follows:

[0076] ;

[0077] in, Indicates global enhancement features, This represents the fusion coefficient, with a value of 0.25. Represents the fusion weight matrix. Indicates a broadcast extension operation. This represents the transpose operation of the projection matrix.

[0078] Furthermore, the global fusion weight matrix in step S423 is generated as follows: for the location of the CLS token feature, the global fusion weight is the robust similarity; for the location of the image patch feature, the global fusion weight is the product of the robust similarity and the spatial saliency value of the corresponding location; the calculation formula is:

[0079] ;

[0080] in, Represents the position in the fusion weight matrix The weight value, Represents the index in the feature sequence, where Corresponding to the CLS token feature position, Corresponding image patch feature location, This represents the average saliency map extracted from all candidate masks.

[0081] Specifically, the average saliency map is generated through the following steps: for each candidate mask, the average brightness of the pixels in its region is calculated as the saliency value of the candidate mask; the saliency values ​​of each candidate mask are assigned to the spatial locations covered by the corresponding candidate mask to form the saliency map of each candidate mask; the saliency maps of all candidate masks are arithmetically averaged to obtain the average saliency map.

[0082] Furthermore, in step S1, the local input image is constructed by multiplying the original image element-wise with each candidate mask to obtain the local input image corresponding to each candidate mask; the global input image is constructed by performing Gaussian blur processing on the areas of the original image not covered by the candidate masks to obtain the global input image with overall context preserved.

[0083] Furthermore, the multi-granularity feature extraction in step S2 includes: using CLIP text encoder to extract full sentence embeddings as global text embeddings; using natural language parsing tools to extract noun phrases in the referential text, and using CLIP text encoder to encode them as phrase embeddings as local text embeddings.

[0084] Furthermore, in step S3, both the local adapter and the global adapter are training-free modules, directly embedded into the Transformer module of the pre-trained dual-branch CLIP image encoder with frozen parameters.

[0085] Another embodiment of the present invention provides an electronic device including a memory and a processor, wherein the memory stores computer-readable instructions, and when the instructions are executed by the processor, the processor performs the above-described zero-sample reference image segmentation method.

[0086] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can execute instructions to perform the above-described zero-sample reference image segmentation method.

[0087] In this embodiment, three publicly available segmentation datasets, RefCOCO, RefCOCO+, and RefCOCOg, are used for performance verification. The evaluation metrics are the overall Intersection over Union (oIoU) and the average Intersection over Union (mIoU). The experimental setup is as follows: Image size: Input images are uniformly adjusted to 384×384 pixels; Image encoder: ViT-B / 16 image encoder pre-trained with CLIP; Text encoder: ViT-B / 16 text encoder pre-trained with CLIP; Candidate mask generator: SAM-ViT-H pre-trained model; Adapter insertion position: Local and global adapters are embedded in the 6th, 9th, and 12th layers of the CLIP image encoder's Transformer module; Local and global adapters are inserted after the feedforward network layer and before the residual connection of the Transformer module; In the same Transformer module layer, local and global adapters are inserted in parallel, processing the visual features of the local and global perception branches respectively, and the two adapters work independently within the layer; Hyperparameter settings: γ=0.2, β=0.25, α=0.4.

[0088] Experimental results show that, under zero-sample settings, the proposed method achieves a mean intersection-union ratio (mIoU) of 50.76% on the RefCOCO validation set, significantly outperforming existing state-of-the-art methods; it achieves a mean intersection-union ratio (mIoU) of 50.29% on the RefCOCO+ test set, demonstrating excellent robustness; and it achieves a mean intersection-union ratio (mIoU) of 51.68% on the RefCOCOg test set, proving the effectiveness of the proposed method in handling complex language descriptions.

[0089] This method significantly outperforms state-of-the-art zero-shot methods, such as TAS and HybridGL, on three publicly available referential segmentation datasets: RefCOCO, RefCOCO+, and RefCOCOg, validating the effectiveness and robustness of the invention.

[0090] Table 1. Performance comparison of overall intersection-union ratio (oIoU) on the segmentation datasets RefCOCO, RefCOCO+, and RefCOCOg.

[0091]

[0092] Table 2. Performance comparison of mean intersection-union ratio (mIoU) on the segmentation datasets RefCOCO, RefCOCO+, and RefCOCOg.

[0093]

[0094] The advantages of this method are: no training required: the adapter module is designed to be training-free and is directly embedded in the frozen encoder, saving computational resources; multimodal early interaction: deep fusion of vision and text is achieved during the encoding stage to avoid feature misalignment; dual-branch complementarity: the local adapter enhances detail perception, the global adapter improves context robustness, and the two work together to improve segmentation accuracy.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A zero-shot reference image segmentation method based on a text-aware adapter, characterized in that, Includes the following steps: Step S1: Collect the original image to be segmented and the referential text, and use the pre-trained segmentation model to generate a candidate mask set for the original image. Based on the candidate mask set, construct the local input image and the global input image respectively. Step S2: Perform multi-granularity feature extraction on the referential text, extracting global text embeddings at the whole sentence level and local text embeddings at the phrase level respectively; Step S3: Construct a dual-branch CLIP image encoder containing a local perception branch and a global perception branch, keep the parameters of the dual-branch CLIP image encoder frozen, and embed the training-free local adapter and global adapter in the Transformer module of the dual-branch CLIP image encoder. Step S4: In the local perception branch, the local input image is input into the CLIP image encoder for extraction to obtain regional visual features; the local adapter is used to fuse the local text embedding with the regional visual features, and fine-grained aligned local enhancement features are generated through weighted regional feature aggregation and gating fusion mechanism. In the global perception branch, the global input image is input into the CLIP image encoder for extraction to obtain global visual features; By using a global adapter to fuse global text embeddings with global visual features, and through variance-based feature enhancement and robust similarity calculation, global enhanced features with noise resistance are generated. Step S5: In the inference stage, local enhancement features and global enhancement features are fused. The semantic similarity score between each candidate mask in the candidate mask set and the referential text is calculated using the fused features. The candidate mask with the highest score is selected as the final referential image segmentation result, thus completing the zero-shot referential image segmentation task. The specific processing procedure for the local sensing branch is as follows: Step S411: Spatial downsampling and flattening are performed on the candidate mask to obtain a candidate mask vector aligned with the visual features of the region; Step S412: Based on the aligned candidate mask vectors, perform weighted aggregation of the region visual features, and extract the region visual features corresponding to each candidate mask. The calculation formula is as follows: ; in, Indicates the first The visual features of the regions corresponding to each candidate mask, where L represents the total number of image patches. Indicates the first Image blocks, Indicates the first The candidate mask in the th... Weights on each image patch Represents the first in the visual feature sequence Feature vectors of image patches Represents the numerical stability constant; Step S413: Project the visual features of the regions corresponding to each candidate mask onto the text feature space, and calculate their semantic similarity with the local text embedding; Step S414: Generate a position weight matrix based on semantic similarity, and inject local text embeddings into the visual feature sequence using a gated fusion mechanism to obtain local enhanced features; The specific implementation of the gating fusion mechanism in step S414 is as follows: gating weights are generated using the Sigmoid activation function, and local enhanced features are calculated according to the following formula: ; in, Indicates local enhancement features, Represents a sequence of visual features. Represents the fusion coefficient. Indicates the gating weight, This represents the position weight matrix based on semantic similarity. This represents the expanded text feature vector.

2. The zero-shot referential image segmentation method based on a text-aware adapter according to claim 1, characterized in that: The processing procedure for the global awareness branch is as follows: Step S421: Calculate the weighted global features based on the channel variance of image patch features in the global visual features, and then perform weighted fusion with the CLS token features to obtain the enhanced global visual features. The calculation formula is as follows: ; in, This represents the enhanced global visual features. Represents the balance coefficient. Indicates the characteristics of the CLS token. This represents the natural exponential function. Indicates the first Channel variance of features of an image patch The parameter represents temperature, and k represents the index variable in the summation process. Represents the feature vector of the k-th image patch Variance in the channel dimension; Step S422: Project the enhanced global visual features onto the text feature space and use the hyperbolic tangent function to calculate its robust similarity with the global text embedding; Step S423: Generate a global fusion weight matrix by combining robust similarity and spatial saliency maps; Step S424: Inject global text embedding into global visual features based on the global fusion weight matrix to obtain global enhanced features. The calculation formula is as follows: ; in, Indicates global enhancement features, Represents the fusion coefficient. Represents the fusion weight matrix. Indicates a broadcast extension operation. Indicates global text embedding. This represents the transpose operation of the projection matrix.

3. The zero-shot referential image segmentation method based on a text-aware adapter according to claim 2, characterized in that: In step S423, the global fusion weight matrix is ​​generated as follows: for the location of the CLS token feature, the global fusion weight is the robust similarity; for the location of the image patch feature, the global fusion weight is the product of the robust similarity and the spatial saliency value of the corresponding location.

4. The zero-shot referential image segmentation method based on a text-aware adapter according to claim 1, characterized in that: The local input image in step S1 is constructed by multiplying the original image with each candidate mask element by element to obtain the local input image corresponding to each candidate mask. The global input image is constructed by performing Gaussian blur processing on the areas of the original image not covered by the candidate masks to obtain the global input image with overall context preservation.

5. The zero-shot referential image segmentation method based on a text-aware adapter according to claim 1, characterized in that: Step S2, multi-granularity feature extraction, includes: using CLIP text encoder to extract full sentence embeddings as global text embeddings; using natural language parsing tools to extract noun phrases from referential texts, and using CLIP text encoder to encode them as phrase embeddings as local text embeddings.

6. The zero-shot referential image segmentation method based on a text-aware adapter according to claim 1, characterized in that: In step S3, both the local adapter and the global adapter are training-free modules, which are directly embedded into the Transformer module of the pre-trained dual-branch CLIP image encoder with frozen parameters.

7. An electronic device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that, When the instruction is executed by the processor, the processor implements the zero-sample referential image segmentation method according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the zero-sample referential image segmentation method as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Zero sample reference image segmentation method based on hierarchical prompt and directional clue

    CN119049057A

  • CLIP-based single-stage zero-sample semantic segmentation method and apparatus

    CN119693649A