An open-vocabulary semantic segmentation method based on diffusion model

By optimizing attention information fusion and autonomous target category extraction using a diffusion model, the problem of insufficient fine recognition capability in existing methods is solved, achieving high-precision and low-cost open-vocabulary semantic segmentation, which is applicable to target segmentation of any category in images.

CN120953614BActive Publication Date: 2026-05-19BEIJING INFORMATION SCI & TECH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
BEIJING INFORMATION SCI & TECH UNIV
Filing Date
2025-08-04
Publication Date
2026-05-19

AI Technical Summary

Technical Problem

Existing open-vocabulary semantic segmentation methods rely on the visual language pre-trained model CLIP, which lacks the ability to finely identify specific locations in images and requires additional training and supervision information, resulting in high costs and poor generalization.

Method used

By optimizing attention information fusion using a diffusion model, high-precision semantic segmentation is achieved through hierarchical attention fusion and autonomous target category set extraction, combined with end-padding character embedding adjustment and target category embedding enhancement.

Benefits of technology

It achieves high-precision pixel-level semantic segmentation, has open vocabulary characteristics, requires no additional training and supervision, reduces implementation costs, and expands the scope of application and flexibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120953614B_ABST
    Figure CN120953614B_ABST
Patent Text Reader

Abstract

The application discloses an open vocabulary semantic segmentation method based on a diffusion model, comprising the following steps: according to an input image, fusing global semantics and local semantics of the image and automatically extracting a target class set and a text prompt in the image; for each target class, processing the target class and the text prompt based on a tail filler embedding adjustment strategy and a target class embedding enhancement method to obtain an optimized text embedding; for each target class, obtaining a target class activation map through hierarchical attention fusion of the diffusion model according to the optimized text embedding; and post-processing each target class activation map to fuse and generate a final semantic segmentation result. The method can generate a high-precision segmentation mask according to the input image; and secondly, due to the open vocabulary characteristic, the model can extract any class in the image and generate a corresponding segmentation result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, and particularly relates to an open-vocabulary semantic segmentation method based on a diffusion model. Background Technology

[0002] Open-vocabulary semantic segmentation is a visual task that has emerged in recent years, aiming to segment objects of any category in an image at the pixel level. Unlike traditional semantic segmentation, which can only handle predefined fixed categories, this task has the ability to segment arbitrary object categories, thus breaking through category limitations and having significant implications for generalization and practicality.

[0003] Classical open-vocabulary semantic segmentation methods often rely on the visual language pre-training model CLIP (Contrastive Language-Image Pre-training), a model trained on large-scale image-text pairs that can align images with text. However, this learning approach, which focuses on overall matching, lacks the ability to finely identify specific locations within an image, making it unsuitable for pixel-level accuracy semantic segmentation tasks. Some methods attempt to compensate for this shortcoming by adding extra training or supervision information to CLIP, but this also incurs additional overhead.

[0004] In recent years, diffusion models have achieved significant breakthroughs in image generation and, due to their rich semantic information, are increasingly being applied in other fields. However, existing methods using diffusion models for segmentation simply extract attention information during the generation process, lacking consideration for how to optimize and utilize this information. Some of these works also require additional training and supervision information, or the pre-construction of a large knowledge base to assist in reasoning.

[0005] Given the shortcomings of existing methods, there is an urgent need to propose an open-vocabulary semantic segmentation method based on a diffusion model. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an open-vocabulary semantic segmentation method based on a diffusion model. This method leverages the diffusion model's precise understanding of object shape, structure, texture, and contextual relationships. During the diffusion model's denoising process, it optimizes and extracts attention information and fuses this attention information in a more reasonable manner, using this as the basis for semantic segmentation. Simultaneously, a fully autonomous target category set extraction method is proposed, thereby achieving more refined masks while exhibiting low cost.

[0007] This invention proposes an open-vocabulary semantic segmentation method based on a diffusion model, comprising the following steps:

[0008] Based on the input image, the global and local semantics of the image are fused, and the target category set and text prompts in the image are automatically extracted;

[0009] For each target category, the target category and text prompt are processed based on the trailing padding embedding adjustment strategy and the target category embedding enhancement method to obtain the optimized text embedding;

[0010] For each target category, based on the optimized text embedding, the diffusion model obtains the target category activation map through hierarchical attention fusion;

[0011] The activation maps for each target category are post-processed and fused to generate the final semantic segmentation result.

[0012] Optionally, the process of fusing the global and local semantics of the image based on the input image and automatically extracting the target category set and text prompts in the image includes:

[0013] The input image is fed into a visual Transformer network to extract semantic features, and pixel clustering is performed based on these features to obtain several local segmentation regions. For each local segmentation region, pixels within the smallest bounding rectangle are extracted to obtain a local image. The input image and the local image are then fed into a visual-language model to generate descriptive text. Natural language processing tools are used to extract nouns from the descriptive text, and these nouns are converted into feature vectors. The similarity between feature vectors is calculated, and only one noun among several with a similarity higher than a preset threshold is retained, ultimately forming a target category set. The categories in the target category set are used as prompt information and are again fed into the visual-language model to generate text prompts that include all target categories.

[0014] Optionally, the process of processing the target category and text prompt based on the trailing padding embedding adjustment strategy and the target category embedding enhancement method for each target category to obtain the optimized text embedding includes:

[0015] The original text prompt is input into the text encoder to obtain the original text embedding; for the target category, a simple text prompt is constructed and input into the text encoder to obtain the simple text embedding; a trailing padding embedding adjustment strategy is adopted to weight and combine the trailing padding embeddings in the original text embedding and the trailing padding embeddings in the simple text embedding; at the same time, the difference between the target category embedding and the global context information embedding is amplified by the target category embedding enhancement method to obtain the optimized text embedding.

[0016] Optionally, the process by which the diffusion model obtains the target category activation map through hierarchical attention fusion for each target category based on the optimized text embedding includes:

[0017] Noise is added to the input image, and under the guidance of optimized text embedding, the diffusion model denoises the input image, generating cross-attention maps and self-attention maps at different resolution levels. The cross-attention maps are then fused hierarchically from coarse to fine, with the lowest-level cross-attention map upsampled to the target level and weighted and summed with the target level's cross-attention map to obtain the fused cross-attention map. The fused cross-attention map is then refined using self-attention maps of the corresponding resolution to obtain the hybrid attention map at each scale. Through a multi-level consistency check mechanism, the hybrid attention maps of all levels are upsampled to the maximum resolution, and the minimum value is taken pixel by pixel to generate the activation map of the target category.

[0018] Optionally, the process of post-processing the activation maps of each target category and fusing them to generate the final semantic segmentation result includes:

[0019] The activation map for each target category is upsampled to the resolution of the input image and refined using a conditional random field to obtain a semantic segmentation mask for each target category. The semantic segmentation masks for all target categories are then merged to obtain the final semantic segmentation result for the entire image.

[0020] Optionally, the visual Transformer network is a ViT network, the visual-language model is a BLIP-2 model, the natural language processing tool is the spaCy library, and the text encoder is the CLIP text encoder.

[0021] Optionally, the conditional random field is a DenseCRF.

[0022] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0023] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0024] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method.

[0025] Compared with the prior art, the present invention has the following advantages and technical effects:

[0026] High-precision semantic segmentation: This invention automatically extracts target category sets and text prompts from images by fusing global and local semantic information, providing a precise semantic foundation for subsequent semantic segmentation. Simultaneously, text embedding optimization techniques based on end-padding embedding adjustment and target category embedding enhancement significantly improve the quality of attention maps during diffusion model denoising, resulting in more accurate target category activation maps. This achieves high-precision pixel-level semantic segmentation of targets of any category in an image.

[0027] Open vocabulary characteristic: Since this invention does not rely on predefined fixed categories, but achieves semantic segmentation by automatically extracting the target category set and text prompts from the image, it possesses the characteristic of an open vocabulary. This means that the model can flexibly handle any category of target appearing in the image without prior training or labeling of these categories, greatly expanding the application scope and flexibility of semantic segmentation.

[0028] Efficient utilization of attention information: This invention fully utilizes the cross-attention maps and self-attention maps generated by the diffusion model at different resolution levels through a hierarchical attention fusion strategy. This fusion method not only preserves the robust semantic localization information at the low level, but also integrates the detailed and contour information at the high level. At the same time, it effectively suppresses false positive activations through a multi-level consistency check mechanism, thereby generating high-quality target category activation maps and further improving the accuracy and reliability of semantic segmentation.

[0029] No additional training or supervision required: This invention eliminates the need for additional training, supervision, or knowledge base construction, directly utilizing the semantic understanding and attention mechanisms of the diffusion model for semantic segmentation. This significantly reduces the implementation cost and complexity of semantic segmentation, improves the model's generalization ability and practicality, and enables it to quickly adapt to semantic segmentation tasks across different scenarios and target categories. Attached Figure Description

[0030] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0031] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram illustrating text embedding optimization according to an embodiment of the present invention;

[0033] Figure 3 This is a cross-attention map corresponding to different words in an embodiment of the present invention;

[0034] Figure 4This is a cross-attention graph of the word "cat" at different levels in an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of hierarchical attention fusion according to an embodiment of the present invention;

[0036] Figure 6 This is a schematic diagram of the generated category activation map and semantic segmentation result in an embodiment of the present invention, wherein (a) is two input images, (b) is the activation map of the first target category, (c) is the activation map of the second target category, and (d) is a schematic diagram of the final segmentation result of the input image. Detailed Implementation

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0039] Example 1

[0040] Traditional semantic segmentation requires collecting a large number of images and performing pixel-level manual annotation. It can only handle target categories that have appeared in the training set, resulting in high costs and weak generalization. Open-vocabulary semantic segmentation aims to perform pixel-level segmentation of objects of any category in an image, breaking through category limitations and having significant implications for generalization and practicality. Existing open-vocabulary semantic segmentation methods often rely on the visual language pre-trained model CLIP, lacking the ability to finely identify specific locations in the image, which is not conducive to pixel-level semantic segmentation. In view of the shortcomings of existing methods, this embodiment provides an open-vocabulary semantic segmentation method based on a diffusion model. This method optimizes text embedding during diffusion model denoising and integrates multi-level attention, enabling higher-precision semantic segmentation.

[0041] like Figure 1 As shown, this embodiment specifically includes the following steps:

[0042] Based on the input image, the global and local semantics of the image are fused, and the target category set and text prompts in the image are automatically extracted;

[0043] For each target category, the target category and text prompt are processed based on the trailing padding embedding adjustment strategy and the target category embedding enhancement method to obtain the optimized text embedding;

[0044] For each target category, based on the optimized text embedding, the diffusion model obtains the target category activation map through hierarchical attention fusion;

[0045] The activation maps for each target category are post-processed and fused to generate the final semantic segmentation result.

[0046] An implementable process for fusing global and local semantics of an input image and automatically extracting a set of target categories and text prompts from the image includes:

[0047] The input image is fed into a visual Transformer network to extract semantic features, and pixel clustering is performed based on these features to obtain several local segmentation regions. For each local segmentation region, pixels within the smallest bounding rectangle are extracted to obtain a local image. The input image and the local image are then fed into a visual-language model to generate descriptive text. Natural language processing tools are used to extract nouns from the descriptive text, and these nouns are converted into feature vectors. The similarity between feature vectors is calculated, and only one noun among several with a similarity higher than a preset threshold is retained, ultimately forming a target category set. The categories in the target category set are used as prompt information and are again fed into the visual-language model to generate text prompts that include all target categories.

[0048] Furthermore, the visual Transformer network is a ViT network, the visual-language model is a BLIP-2 model, the natural language processing tool is the spaCy library, and the text encoder is the CLIP text encoder.

[0049] The feasible process of processing the target category and text prompt based on the trailing padding embedding adjustment strategy and the target category embedding enhancement method for each target category to obtain the optimized text embedding includes:

[0050] The original text prompt is input into the text encoder to obtain the original text embedding; for the target category, a simple text prompt is constructed and input into the text encoder to obtain the simple text embedding; a trailing padding embedding adjustment strategy is adopted to weight and combine the trailing padding embeddings in the original text embedding and the trailing padding embeddings in the simple text embedding; at the same time, the difference between the target category embedding and the global context information embedding is amplified by the target category embedding enhancement method to obtain the optimized text embedding.

[0051] Implementable, the process by which the diffusion model obtains the target category activation map through hierarchical attention fusion for each target category based on the optimized text embedding includes:

[0052] Noise is added to the input image, and under the guidance of optimized text embedding, the diffusion model denoises the input image, generating cross-attention maps and self-attention maps at different resolution levels. The cross-attention maps are then fused hierarchically from coarse to fine, with the lowest-level cross-attention map upsampled to the target level and weighted and summed with the target level's cross-attention map to obtain the fused cross-attention map. The fused cross-attention map is then refined using self-attention maps of the corresponding resolution to obtain the hybrid attention map at each scale. Through a multi-level consistency check mechanism, the hybrid attention maps of all levels are upsampled to the maximum resolution, and the minimum value is taken pixel by pixel to generate the activation map of the target category.

[0053] The feasible process of post-processing the activation maps of each target category and fusing them to generate the final semantic segmentation result includes:

[0054] The activation map for each target category is upsampled to the resolution of the input image and refined using a conditional random field to obtain a semantic segmentation mask for each target category. The semantic segmentation masks for all target categories are then merged to obtain the final semantic segmentation result for the entire image.

[0055] Furthermore, the conditional random field is a DenseCRF.

[0056] Example 2

[0057] The overall process of this embodiment is as follows:

[0058] By fusing global and local semantics of an image, and extracting target category sets and text cues from the image, the following steps are specifically included:

[0059] For open-vocabulary semantic segmentation, the first step is to determine which target categories are present in the image.

[0060] 1) Input an image I, and feed the entire image I into the ViT network to extract semantic features. Based on the semantic features, perform pixel clustering to obtain K local segmentation regions {R}. k}(k=1..K). For each local segmentation region R k Extracting pixels from its smallest bounding rectangle region yields the local image I. k .

[0061] 2) Combine the entire image I and the local image {I} kThe text (k = 1..K) is sequentially input into the visual-language model BLIP-2 to generate text describing the entire image and local images. The spaCy library is used to extract the nouns from these texts. The CLIP text encoder converts these nouns into feature vectors and calculates the similarity between the feature vectors. Only one of the nouns with a similarity between the feature vectors higher than a preset threshold is retained to form the target category set N in the image, for example, N = {"dog", "flower"}.

[0062] 3) Using all categories in the target category set N as cue information, input the entire image I back into the visual-language model BLIP-2 to generate the original text cue P containing all categories in the target category set N. raw , such as "A photo of dog smelling flower".

[0063] For each target category, the text embedding is optimized based on trailing padding embedding adjustments and target category embedding enhancements. This includes the following steps:

[0064] Original text prompt P raw After being fed into the text encoder, the original text embedding E is obtained. raw As shown in formula (1).

[0065]

[0066] in, Represents a text encoder, e sot and e eot This represents the embedded representation of the start-of-text (SOT) character and the end-of-text (EOT) character after encoding. tn Indicates the original text prompt P raw The embedding of the nth word in the set N is c, which is a target category to be segmented (e.g., "dog") in the target category set N. c This represents the embedding of the target category c. eot It contains global context information for categories other than target category c, which interferes with the segmentation of target category c.

[0067] To avoid e eot This interferes with segmentation of target category c. For target category c, a simple text prompt P is constructed. simple The format is "A photo of c", and it also uses a text encoder. Encoding it yields a simple text embedding E simple As shown in formula (2).

[0068]

[0069] Then, in order to minimize e eot The inclusion of information other than target category c interferes with the segmentation of target category c. Therefore, an EOT embedding adjustment strategy is adopted, i.e., E... raw e eot Replace with e eot With e′ eot Weighted combination, i.e. w·e eot +(1-w)·e′ eot , where w represents the weight parameter.

[0070] To further enhance the role of target category c, this embodiment also employs a target category embedding enhancement method, selectively amplifying the target category embedding e. c With e containing global context information eot The differences between them are shown in formula (3).

[0071]

[0072] Where λ is a hyperparameter controlling the enhancement intensity, and p represents e eot The length.

[0073] After EOT embedding adjustment and target category embedding enhancement, the final optimized text embedding E is obtained. refine As shown in Equation (4), it is used as a conditional input in the denoising U-Net of the diffusion model.

[0074]

[0075] Figure 2 A concrete example is shown. Based on the original text prompt P raw =“A photo of a dog smelling a flower”, the target category to be segmented is c = "dog", generate a simple hint P simple =“A photo of a dog”, the two prompts are fed into the text encoder to obtain the text embedding E respectively. raw and E simple Then, E raw e eot Replace with e eot With e′ eot Weighted combination, i.e. w·e eot +(1-w)·e′ eot Simultaneously, embedding enhancement is performed on the target category c, ultimately yielding the optimized text embedding E. refine .

[0076] For each target category, based on the optimized text embedding, a diffusion model is used for denoising, and a target category activation map is obtained based on hierarchical attention fusion. This process includes the following steps:

[0077] Add noise to the input image I, and then optimize the text embedding E for the target class c. refine Under guidance, the diffusion model progressively denoises and recovers the input image I. During the denoising process, two types of attention information are used: cross-attention models image features and text embeddings, while self-attention models different spatial relationships. These attention levels also exist at different resolutions: 8, 16, 32, and 64. The cross-attention map and self-attention map of the l-th layer are represented as follows: and (l = 8, 16, 32, 64). The lower the attention level, the more attention is paid to the position of the target object; the higher the level, the more attention is paid to the outline and details of the object. Figure 3 and Figure 4 The semantic information contained in cross-attention and the different regions focused on by attention at different levels are shown respectively. In order to make full use of attention maps of different types and levels, this embodiment designs a hierarchical attention fusion strategy.

[0078] To perform a coarse-to-fine hierarchical fusion of cross-attention maps at different resolution levels, we first perform a coarse-to-fine fusion of the lowest level... Upsampled to 16 and compared with the cross-attention map The weighted summation yields the fused attention map. As shown in formula (5).

[0079]

[0080] Among them, U ×2 (·) represents upsampling at twice the resolution, · represents dot product, ω l This represents the weight coefficient of each layer. This process is iterated continuously, upsampling the result of the fusion of the lower layer and performing weighted fusion with the cross-attention map of the current layer, as shown in formula (6).

[0081]

[0082] In this way, low-level, robust semantic location information is gradually propagated and integrated into the high-level graph.

[0083] After obtaining the fused multi-scale cross-attention map, the self-attention map of the corresponding resolution is used. Refine it to obtain the hybrid attention map at each scale. As shown in formula (7).

[0084]

[0085] in This represents pixel-wise multiplication. This operation can filter out semantically related but structurally discontinuous regions, thereby reducing holes within the attention map while making the boundaries sharper.

[0086] Finally, using a multi-level consistency check mechanism, the mixed attention maps of all levels are upsampled to the maximum resolution of 64, and the minimum value of the mixed attention maps of different levels is taken as the activation map A of the target class c. final As shown in formula (8).

[0087]

[0088] Among them U ↑64 This indicates upsampling to 64, and `min` means taking the minimum value pixel by pixel. This operation requires the attention map to show high activation at each level to be considered part of the target object, which can effectively suppress false positive activations that occur at a single level.

[0089] Figure 5 A schematic diagram illustrating hierarchical attention fusion is shown. The image and optimized text are embedded into E... refine After being fed into the denoising network of the diffusion model, the cross-attention and self-attention maps are extracted during the denoising process. The cross-attention maps are then hierarchically weighted and fused with their corresponding self-attention maps. The fused results are upsampled to the same level, and the minimum value is taken pixel by pixel to obtain the final attention map.

[0090] Post-processing is performed on the activation maps of each target category, and the maps are fused to generate the final semantic segmentation result. The specific steps include:

[0091] After obtaining the activation map A for each target category c final Then, the image is upsampled to the resolution of the input image I and refined using DenseCRF to obtain the semantic segmentation mask for target category c. The semantic segmentation masks for all target categories are then merged to obtain the final semantic segmentation result for the entire image. Figure 6 The following shows some of the segmentation results obtained using this embodiment.

[0092] Example 3

[0093] This embodiment also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0094] Example 4

[0095] This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0096] Example 5

[0097] This embodiment also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0098] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An open-vocabulary semantic segmentation method based on a diffusion model, characterized in that, Includes the following steps: Based on the input image, the global and local semantics of the image are fused, and the target category set and text prompts in the image are automatically extracted; For each target category, the target category and text prompt are processed based on the trailing padding embedding adjustment strategy and the target category embedding enhancement method to obtain the optimized text embedding; For each target category, based on the optimized text embedding, the diffusion model obtains the target category activation map through hierarchical attention fusion; The activation maps for each target category are post-processed and fused to generate the final semantic segmentation result; The process of fusing global and local semantics of the input image and automatically extracting the target category set and text prompts from the image includes: The input image is fed into a visual Transformer network to extract semantic features, and pixel clustering is performed based on these features to obtain several local segmentation regions. For each local segmentation region, pixels within the smallest bounding rectangle are extracted to obtain a local image. The input image and the local image are then fed into a visual-language model to generate descriptive text. Natural language processing tools are used to extract nouns from the descriptive text, and these nouns are converted into feature vectors. The similarity between feature vectors is calculated, and only one noun among several with a similarity higher than a preset threshold is retained, ultimately forming a target category set. The categories in the target category set are used as prompt information and are again fed into the visual-language model to generate text prompts that include all target categories. The process of processing the target category and text prompt based on the trailing padding embedding adjustment strategy and the target category embedding enhancement method for each target category to obtain the optimized text embedding includes: The original text prompt is input into the text encoder to obtain the original text embedding; for the target category, a simple text prompt is constructed and input into the text encoder to obtain the simple text embedding; the end padding embedding in the original text embedding and the end padding embedding in the simple text embedding are weighted and combined using the end padding embedding adjustment strategy; at the same time, the difference between the target category embedding and the global context information embedding is amplified by the target category embedding enhancement method to obtain the optimized text embedding. The process by which the diffusion model obtains the target category activation map for each target category through hierarchical attention fusion based on the optimized text embedding includes: Noise is added to the input image, and under the guidance of optimized text embedding, the diffusion model denoises the input image, generating cross-attention maps and self-attention maps at different resolution levels. The cross-attention maps are then fused hierarchically from coarse to fine, with the lowest-level cross-attention map upsampled to the target level and weighted and summed with the target level's cross-attention map to obtain the fused cross-attention map. The fused cross-attention map is then refined using self-attention maps of the corresponding resolution to obtain the hybrid attention map at each scale. Through a multi-level consistency check mechanism, the hybrid attention maps of all levels are upsampled to the maximum resolution, and the minimum value is taken pixel by pixel to generate the activation map of the target category.

2. The method according to claim 1, characterized in that, The process of post-processing the activation maps of each target category and fusing them to generate the final semantic segmentation result includes: The activation map of each target category is upsampled to the resolution of the input image and refined using a conditional random field to obtain the semantic segmentation mask for each target category. The semantic segmentation masks of all target categories are then merged to obtain the final semantic segmentation result of the entire image.

3. The method according to claim 1, characterized in that, The visual Transformer network is the ViT network, the visual-language model is the BLIP-2 model, the natural language processing tool is the spaCy library, and the text encoder is the CLIP text encoder.

4. The method according to claim 2, characterized in that, The conditional random field is a DenseCRF.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-4.

7. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-4.