A weakly supervised semantic segmentation method and system based on text mask collaborative prompting

Abstract

The present invention discloses a weakly supervised semantic segmentation method and system based on text-mask collaborative prompting. The method comprises: obtaining image information and text information to be processed; inputting the image information to be processed into a weakly supervised semantic segmentation model for processing, and outputting a segmentation result. The weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using a self-supervised text-prompted STP branch structure and a weakly supervised mask-prompted WMP branch structure. The self-supervised text-prompted STP branch structure utilizes cross-layer block association and multi-scale text features to suppress background noise, while the weakly supervised mask-prompted WMP branch structure uses mask seeds to refine boundary accuracy. The present invention enhances CAM performance by collaboratively using text and mask prompts.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a weakly supervised semantic segmentation method and system based on text mask collaborative prompting. Background Art

[0002] Semantic segmentation plays a crucial role in computer vision applications, including medical imaging, remote sensing, and autonomous driving. However, the high cost of pixel-level annotations limits its scalability. Consequently, the more challenging and realistic weakly supervised semantic segmentation (WSSS) paradigm has recently attracted significant attention. WSSS aims to reduce manual annotation workload by leveraging sparse, low-cost annotations (e.g., bounding boxes, scribbles, and image-level class labels) to train semantic segmentation models. Among these forms of supervision, image-level annotations are the most popular due to their easy availability through web crawlers. Recent advances in cue fine-tuning have demonstrated significant adaptability and performance across a variety of computer vision tasks. However, because classification labels only provide coarse-grained information about specific categories at the image level, semantic segmentation (CAM) often highlights key regions while being affected by background noise. This limitation results in suboptimal pseudo-labels that fail to fully capture the extent of the target object. Therefore, recent research on WSSS-based on-the-fly fine-tuning has focused on better aligning visual concepts in images with corresponding textual labels in open-world settings. For example, clarity-based cue selection has been proposed to obtain contextual cues that positively impact activation performance. Others have found that modifying textual cues with class labels can significantly impact CAM. Some have utilized coarse-to-fine contextual cues to gradually refine the initial class activation map. Nevertheless, textual cues alone have limitations in WSSS because they lack fine-grained localization, which makes it challenging to solve problems such as ambiguous object boundaries and high inter-class similarity.

[0003] Recent advances in cue adaptation have demonstrated significant adaptability and performance across a variety of computer vision tasks. However, because classification labels only provide coarse-grained information about specific categories at the image level, Cam (Cam) often highlights key regions while being affected by background noise. This limitation results in suboptimal pseudo-labels that fail to fully capture the extent of the target object. Therefore, recent research in WSSS based on on-the-fly adaptation has focused on better aligning visual concepts in images with corresponding textual labels in open-world settings. For example, Lin et al. proposed clarity-based cue selection to obtain contextual cues that positively impact activation performance. Balamurali Murugesan et al. found that modifying textual cues with class labels can significantly improve Cam. Similarly, Yang et al. utilized coarse-to-fine contextual cues to gradually refine the initial class activation map. Nevertheless, textual cues alone have limitations in WSSS due to their lack of fine-grained localization, making it challenging to address problems such as ambiguous object boundaries and high inter-class similarity. Despite significant progress in enhancing the category specificity of textual cues, current research still faces challenges with background noise interference and ambiguous target object boundaries. On the one hand, previous studies have attempted to improve class activation maps by modifying either contextual cues or class labels. However, these methods typically operate at a coarse information level and fail to fully utilize the multi-scale semantic information inherent in text features. As a result, the CAMs generated by existing methods are still susceptible to background noise, resulting in background confounding problems and hindering the effective distinction between foreground and background. On the other hand, the limitations of weak supervision signals in extracting edge features limit the resolution of CAMs in depicting the boundaries of target objects, resulting in ambiguity and inaccuracy in capturing precise object contours. To address these challenges, the present invention proposes a weakly supervised semantic segmentation method and system based on text mask collaborative prompts. Summary of the Invention

[0004] In order to solve the technical problems existing in the above-mentioned prior art, the present invention proposes a weakly supervised semantic segmentation method and system based on text mask collaborative prompting, which reshapes the weakly supervised semantic segmentation pipeline by integrating self-supervised text prompting (STP) and weakly supervised mask prompting (WMP) mechanisms.

[0005] On the one hand, to achieve the above-mentioned object, the present invention provides a weakly supervised semantic segmentation method based on text mask collaborative prompting, comprising:

[0006] Obtain image information and text information to be processed;

[0007] The image information and text information to be processed are input into a weakly supervised semantic segmentation model for processing, and a segmentation result is output; wherein, the weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using a self-supervised text prompt STP branch structure and a weakly supervised mask prompt WMP branch structure; the self-supervised text prompt STP branch structure utilizes cross-layer block association and multi-scale text features to suppress background noise, and the weakly supervised mask prompt WMP branch structure uses mask seeds to refine boundary accuracy.

[0008] Preferably, the self-supervised text prompt STP branch structure includes:

[0009] CAM extraction module: used to extract the initial class activation map CAM through the SoftmaxGradCAM method, and integrate the low-level visual information extracted by TSA in the WMP branch into the Softmax-GradCAM;

[0010] Inter-block cross-layer optimization ICO module: used to derive the self-attention map from the visual encoder for cross-layer averaging, build an affinity map, and capture cross-layer inter-block affinity;

[0011] Cross-modal multi-scale optimization (CMO) module: It is used to extract multi-scale intermediate features from the encoder as queries, and use CAM as keys and values. Through the attention mechanism, it aligns text features with the semantic information in CAM to generate cross-modal feature representations.

[0012] Preferably, the weakly supervised mask hint WMP branch structure includes:

[0013] Weakly supervised seed generation module: used to generate weakly supervised mask seeds based on target area prompts;

[0014] Target Space Awareness (TSA) module: used to perform Gaussian blur processing on the original image using the weakly supervised mask seed and extract spatial information;

[0015] Mask hint MP module: used to fuse and refine the weakly supervised mask seed with the CAM obtained by the STP branch to obtain the final pseudo mask.

[0016] Preferably, the weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using a self-supervised text prompt STP branch structure and a weakly supervised mask prompt WMP branch structure, including:

[0017] First, the spatial information extracted by the target space perception TSA module is injected into the CAM generation stage to obtain the initial CAM;

[0018] Secondly, the cross-modal multi-scale optimization (CMO) module is used to mine multi-scale semantic information in image text information and suppress background noise;

[0019] Finally, the mask hint MP module is used to fuse the weakly supervised mask seeds with the refined CAM to obtain the final pseudo mask, which is used to train a fully supervised semantic segmentation model.

[0020] Preferably, the spatial information extracted by the target space perception TSA module is injected into the CAM generation stage to obtain the initial CAM, including:

[0021] The original image is Gaussian blurred based on the weakly supervised mask seed, and then the spatial information is extracted using the spatial perceptron, specifically:

[0022] f tsa =E tsa (X*G(S wk )+α·Random(X)),

[0023] Where, E tsa is a spatial perceptron, G(·) represents the Gaussian blur kernel, * represents the convolution operation, α is the balance coefficient, Random(·) represents the random enhancement operation, and f tsa is the extracted spatial information, X is the original image information, S wk is the weakly supervised mask seed.

[0024] Preferably, a cross-modal multi-scale optimization (CMO) module is used to mine multi-scale semantic information in image text information to enhance background noise suppression, including:

[0025] Multi-scale intermediate features are extracted from the encoder as queries, and CAM is used as keys and values. Through the attention mechanism, text features are aligned with the semantic information in CAM to generate refined cross-modal feature representations, gradually enhancing CAM areas related to text features while suppressing irrelevant areas.

[0026] Preferably, a mask hint module is used to perform fine boundary supervision in post-processing as follows:

[0027]

[0028] Where P represents the set of foreground classes in the image, λ is a hyperparameter defined based on empirical knowledge, and S wk is the weakly supervised mask seed, is the background mask seed, Q (h，w) is the pixel value at position (h, w), is the mask seed of category c, is the mask seed containing foreground and background.

[0029] Preferably, obtaining the final pseudo mask includes:

[0030]

[0031] Where Norm(·) refers to the minimum-maximum normalization method, θ and μ are hyperparameters set based on empirical knowledge, is the final pseudo mask, is the mask seed containing the foreground and background, is the CAM obtained by TSP branching.

[0032] On the other hand, to achieve the above-mentioned purpose, the present invention also provides a weakly supervised semantic segmentation system based on text mask collaborative prompting, comprising:

[0033] Image information acquisition module: used to obtain image information and text information to be processed;

[0034] Semantic segmentation processing module: used to input the image information and text information to be processed into a weakly supervised semantic segmentation model for processing, and output the segmentation result, wherein the weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using the self-supervised text prompt STP branch structure and the weakly supervised mask prompt WMP branch structure.

[0035] Preferably, the semantic segmentation processing module includes:

[0036] Self-supervised text hint STP branch unit: used to suppress background noise by utilizing cross-layer block association and multi-scale text features;

[0037] Weakly supervised mask hint WMP branch unit: used to refine boundary accuracy using mask seeds.

[0038] Compared with the prior art, the present invention has the following advantages and technical effects:

[0039] This paper addresses challenges such as background noise and edge blur through a collaborative approach that combines text hints with mask hints. The framework proposed in this paper consists of two branches: the STP branch utilizes cross-layer block correlation and multi-scale text features to suppress background noise, while the WMP branch uses mask seeds to refine boundary accuracy. This collaborative optimization significantly enhances the CAM representation to obtain refined pseudo masks for training fully supervised semantic segmentation models. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0041] Figure 1 This is a flowchart of a weakly supervised semantic segmentation method based on text mask collaborative prompting according to an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the workflow of the inter-block cross-layer optimization ICO module according to an embodiment of the present invention;

[0043] Figure 3 Schematic diagram of the workflow of the cross-modal multi-scale optimization (CMO) module according to an embodiment of the present invention;

[0044] Figure 4 Schematic diagram of the mask prompt MP module workflow according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] This embodiment proposes a weakly supervised semantic segmentation method based on text mask collaborative prompts, such as Figure 1 ,include:

[0048] Obtain image information and text information to be processed;

[0049] The image information and text information to be processed are input into a weakly supervised semantic segmentation model for processing, and the segmentation results are output. Among them, the weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using the self-supervised text hint STP branch structure and the weakly supervised mask hint WMP branch structure. The self-supervised text hint STP branch structure uses cross-layer block association and multi-scale text features to suppress background noise, and the weakly supervised mask hint WMP branch structure uses mask seeds to refine boundary accuracy.

[0050] Specifically, the STP branch utilizes cross-layer block correlation and multi-scale text features to suppress background noise, while the WMP branch adopts mask seeds to refine boundary accuracy. This collaborative optimization significantly enhances the CAM representation to obtain refined pseudo masks for training a fully supervised semantic segmentation model.

[0051] Furthermore, the self-supervised text prompt STP branch structure includes:

[0052] CAM extraction module: used to extract the initial class activation map CAM through the SoftmaxGradCAM method, and integrate the low-level visual information extracted by TSA in the WMP branch into the Softmax-GradCAM;

[0053] Inter-block cross-layer optimization ICO module: used to derive the self-attention map from the visual encoder for cross-layer averaging, build an affinity map, and capture cross-layer inter-block affinity;

[0054] Cross-modal multi-scale optimization (CMO) module: It is used to extract multi-scale intermediate features from the encoder as queries, and use CAM as keys and values. Through the attention mechanism, it aligns text features with the semantic information in CAM to generate cross-modal feature representations.

[0055] Specifically, the CAM extraction module uses the SoftmaxGradCAM method to extract the initial CAM. Different from the traditional method of calculating similarity using original image features, the low-level visual information extracted by TSA in the WMP branch is integrated into the Softmax-GradCAM. Specifically, the formula for calculating cosine similarity is logit Y = cos(f I ′,f T ), where f I ′ represents f I and f tsa The weighted sum (f T is the text feature, f I is the image feature). Then the competitive score s of each class is obtained by Softmax c Finally, the initial CAM is derived using the following formula:

[0056]

[0057] Among them, Q (h,w) represents the value at position (h,w), is the weight of the k-th layer feature for category c, N p is the number of patches, For the k-th layer feature of category c, M c is the initial CAM.

[0058] like Figure 2 , the inter-block cross-layer optimization ICO module derives the self-attention map from the L layer of the visual encoder And average them across layers to construct a comprehensive affinity map that captures the affinity between blocks across layers. Using this affinity, the activation map of class c The internal propagation score, thus generating the initial optimized CAM As shown below:

[0059]

[0060] Among them, R represents the shape transformation operation, voc(·) represents the vectorization of the matrix, is the CAM after preliminary refinement, A (l) The first attention layer.

[0061] like Figure 3 , the cross-modal multi-scale optimization CMO module extracts multi-scale intermediate features from the encoder as the query, and uses CAM as the key and value. Through the attention mechanism, the design aligns the text features with the semantic information in the CAM to generate a refined cross-modal feature representation. This method gradually enhances the CAM areas related to the text features while suppressing irrelevant areas. Given the c-class text features of the j-th layer and optimized class activation maps It is possible to compute attention maps that contain semantic information derived from text features As:

[0062]

[0063] Among them, W q 、W k Represent the learnable components, d q Corresponding to the query dimension, in this embodiment, during the initial calculation process,

[0064] Furthermore, the weakly supervised mask prompts the WMP branch structure to include:

[0065] Weakly supervised seed generation module: used to generate weakly supervised mask seeds based on target area prompts;

[0066] Target Space Awareness (TSA) module: used to perform Gaussian blur processing on the original image using the weakly supervised mask seed and extract spatial information;

[0067] Mask hint MP module: It is used to fuse and refine the weakly supervised mask seed with the CAM obtained by the STP branch to obtain the final pseudo mask.

[0068] Specifically, the weakly supervised seed generation module first inputs the original image and its corresponding class label into the pre-trained multimodal large language model KOSMOS, which understands the relationship between image content and class labels and generates target region hint boxes. This target region hint is then provided as the input of the SAM model, which generates an accurate mask based on the target region hint, which is defined as the weakly supervised mask seed S wk .

[0069] The target space perception module (TSA) uses the mask seed S generated by the SAM model wk, based on the mask seed, the original image is Gaussian blurred, and then the spatial perceptron E is used tsa Extracting spatial information, this operation can be expressed by the following equation:

[0070] f tsa =E tsa (X*G(S wk )+α·Random(X)),

[0071] Where, E tsa is a spatial perceptron, G(·) represents the Gaussian blur kernel, * represents the convolution operation, α is the balance coefficient, Random(·) represents the random enhancement operation, and f tsa is the extracted spatial information, X is the original image information, S wk is the weakly supervised mask seed.

[0072] like Figure 4 After the mask prompt module (MP) obtains the seed from the SAM model, it further calculates the background seed using the following formula

[0073]

[0074] Where λ is a hyperparameter defined based on empirical knowledge, is the background mask seed, Q (h，w) is the pixel value at position (h, w), is the mask seed of category c, is the mask seed containing foreground and background.

[0075] Furthermore, the weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using the self-supervised text hint STP branch structure and the weakly supervised mask hint WMP branch structure, including:

[0076] First, the spatial information extracted by the target space perception TSA module is injected into the CAM generation stage to obtain the initial CAM;

[0077] Secondly, the cross-modal multi-scale optimization (CMO) module is used to mine multi-scale semantic information in image text information and suppress background noise;

[0078] Finally, the mask hint MP module is used to fuse the weakly supervised mask seed with the refined CAM to obtain the final pseudo mask, which is used to train a fully supervised semantic segmentation model.

[0079] Specifically, the target space perception TSA module is used to accurately locate the target in the CAM generation stage, and the representation ability of the CAM is enhanced by integrating low-level visual information. The core of TSA is the weakly supervised seed guidance mechanism designed to merge low-level visual information. This enables the model to effectively capture foreground and background information from blurred images. Specifically, the mask seed S generated by the SAM model is used to generate the target space perception TSA module. wk , based on the mask seed, the original image is Gaussian blurred, and then the spatial perceptron E is used tsa Extract spatial information.

[0080] The cross-modal multi-scale optimization (CMO) module is used to deeply mine the multi-scale semantics in the text to enhance background noise suppression. Specifically, multi-scale intermediate features are extracted from the encoder as the query, and CAM is used as the key and value. Through the attention mechanism, the text features are aligned with the semantic information in the CAM to generate a refined cross-modal feature representation. This method gradually enhances the CAM areas related to the text features while suppressing irrelevant areas. Given the c-type text features of the j-th layer and optimized class activation maps It is possible to compute attention maps that contain semantic information derived from text features As:

[0081]

[0082] Among them, W q 、W k Represent the learnable components, d q Corresponding to the query dimension, during the initial calculation process,

[0083] Then, using the rich semantic information To optimize the activation score as follows:

[0084]

[0085] Among them, W v are learnable parameters.

[0086] Specifically, the mask hint module is used to perform fine boundary supervision in the post-processing process to improve the clarity of object boundaries. Specifically, after obtaining the seeds from the SAM model, the background seeds are further calculated using the following formula

[0087]

[0088] In the formula, λ is a hyperparameter defined based on empirical knowledge, S wk is the weakly supervised mask seed, is the background mask seed, Q (h，w) is the pixel value at position (h, w), is the mask seed of category c, is the mask seed containing foreground and background.

[0089] Finally, the weakly supervised seed is fused with the refined CAM to the final pseudo mask using the following formula:

[0090]

[0091] Where Norm(·) refers to the minimum-maximum normalization method, θ and μ are hyperparameters set based on empirical knowledge, is the final pseudo mask, is the mask seed containing the foreground and background, is the CAM obtained by TSP branching.

[0092] This embodiment also provides a weakly supervised semantic segmentation system based on text mask collaborative prompting, including:

[0093] Image information acquisition module: used to obtain image information and text information to be processed;

[0094] Semantic segmentation processing module: used to input the image information and text information to be processed into a weakly supervised semantic segmentation model for processing, and output the segmentation result, wherein the weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using the self-supervised text prompt STP branch structure and the weakly supervised mask prompt WMP branch structure.

[0095] Furthermore, the semantic segmentation processing module includes:

[0096] Self-supervised text hint STP branch unit: used to suppress background noise by utilizing cross-layer block association and multi-scale text features;

[0097] Weakly supervised mask hint WMP branch unit: used to refine boundary accuracy using mask seeds.

[0098] This paper addresses challenges such as background noise and edge blur through a collaborative approach that combines text hints with mask hints. The framework proposed in this paper consists of two branches: the STP branch utilizes cross-layer block correlation and multi-scale text features to suppress background noise, while the WMP branch uses mask seeds to refine boundary accuracy. This collaborative optimization significantly enhances the CAM representation to obtain refined pseudo masks for training fully supervised semantic segmentation models.

[0099] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A weakly supervised semantic segmentation method based on text mask collaborative prompting, characterized by: include: Obtain image information and text information to be processed; Inputting the image information and text information to be processed into a weakly supervised semantic segmentation model for processing, and outputting a segmentation result; wherein the weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using a self-supervised text prompt STP branch structure and a weakly supervised mask prompt WMP branch structure; the self-supervised text prompt STP branch structure utilizes cross-layer block association and multi-scale text features to suppress background noise, and the weakly supervised mask prompt WMP branch structure utilizes mask seeds to refine boundary accuracy; The self-supervised text prompt STP branch structure includes: CAM extraction module: used to extract the initial class activation map CAM through the SoftmaxGradCAM method, and integrate the low-level visual information extracted by TSA in the WMP branch into the Softmax-GradCAM; Inter-block cross-layer optimization ICO module: used to derive the self-attention map from the visual encoder for cross-layer averaging, build an affinity map, and capture cross-layer inter-block affinity; Cross-modal Multi-scale Optimization (CMO) module: This module extracts multi-scale intermediate features from the encoder as queries, uses CAM as keys and values, and uses an attention mechanism to align text features with semantic information in the CAM to generate cross-modal feature representations. The weakly supervised mask prompt WMP branch structure includes: Weakly supervised seed generation module: used to generate weakly supervised mask seeds based on target area prompts; Target Space Awareness (TSA) module: used to perform Gaussian blur processing on the original image using the weakly supervised mask seed and extract spatial information; Mask hint MP module: used to fuse and refine the weakly supervised mask seed with the CAM obtained by the STP branch to obtain the final pseudo mask.

2. The weakly supervised semantic segmentation method based on text mask collaborative prompting according to claim 1 is characterized in that The weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using a self-supervised text prompt STP branch structure and a weakly supervised mask prompt WMP branch structure, including: First, the spatial information extracted by the target space perception TSA module is injected into the CAM generation stage to obtain the initial CAM; Secondly, the cross-modal multi-scale optimization (CMO) module is used to mine multi-scale semantic information in image text information and suppress background noise; Finally, the mask hint MP module is used to fuse the weakly supervised mask seed with the refined CAM to obtain the final pseudo mask, which is used to train a fully supervised semantic segmentation model.

3. The weakly supervised semantic segmentation method based on text mask collaborative prompting according to claim 2 is characterized in that The spatial information extracted by the target space perception TSA module is injected into the CAM generation phase to obtain the initial CAM, which includes: The original image is Gaussian blurred based on the weakly supervised mask seed, and then the spatial information is extracted using the spatial perceptron, specifically: f tsa =E tsa (X*G(S wk )+α·Random(X)), Where, E tsa is a spatial perceptron, G(·) represents the Gaussian blur kernel, * represents the convolution operation, α is the balance coefficient, Random(·) represents the random enhancement operation, and f tsa is the extracted spatial information, X is the original image information, S wk is the weakly supervised mask seed.

4. The weakly supervised semantic segmentation method based on text mask collaborative prompting according to claim 3 is characterized in that The cross-modal multi-scale optimization (CMO) module is used to mine multi-scale semantic information in image text information to enhance background noise suppression, including: Multi-scale intermediate features are extracted from the encoder as queries, and CAM is used as keys and values. Through the attention mechanism, text features are aligned with the semantic information in CAM to generate refined cross-modal feature representations, gradually enhancing CAM areas related to text features while suppressing irrelevant areas.

5. The weakly supervised semantic segmentation method based on text mask collaborative prompting according to claim 4 is characterized in that Use the mask hint module to perform fine boundary supervision in the post-processing process: Where P represents the set of foreground classes in the image, λ is a hyperparameter defined based on empirical knowledge, and S wk is the weakly supervised mask seed, is the background mask seed, Q (h，w) is the pixel value at position (h, w), is the mask seed of category c, is the mask seed containing foreground and background.

6. The weakly supervised semantic segmentation method based on text mask collaborative prompting according to claim 2 is characterized in that Obtaining the final pseudo mask includes: Where Norm(·) refers to the minimum-maximum normalization method, θ and μ are hyperparameters set based on empirical knowledge, is the final pseudo mask, is the mask seed containing the foreground and background, is the CAM obtained by TSP branching.

7. A weakly supervised semantic segmentation system based on text mask collaborative prompting, characterized by: include: Image information acquisition module: used to obtain image information and text information to be processed; Semantic segmentation processing module: used to input the image information and text information to be processed into a weakly supervised semantic segmentation model for processing, and output a segmentation result, wherein the weakly supervised semantic segmentation model is reconstructed and trained by collaboratively using a self-supervised text prompt STP branch structure and a weakly supervised mask prompt WMP branch structure; The self-supervised text prompt STP branch structure includes: CAM extraction module: used to extract the initial class activation map CAM through the SoftmaxGradCAM method, and integrate the low-level visual information extracted by TSA in the WMP branch into the Softmax-GradCAM; Inter-block cross-layer optimization ICO module: used to derive the self-attention map from the visual encoder for cross-layer averaging, build an affinity map, and capture cross-layer inter-block affinity; Cross-modal Multi-scale Optimization (CMO) module: This module extracts multi-scale intermediate features from the encoder as queries, uses CAM as keys and values, and uses an attention mechanism to align text features with semantic information in the CAM to generate cross-modal feature representations. The weakly supervised mask prompt WMP branch structure includes: Weakly supervised seed generation module: used to generate weakly supervised mask seeds based on target area prompts; Target Space Awareness (TSA) module: used to perform Gaussian blur processing on the original image using the weakly supervised mask seed and extract spatial information; Mask hint MP module: used to fuse and refine the weakly supervised mask seed with the CAM obtained by the STP branch to obtain the final pseudo mask.

8. The weakly supervised semantic segmentation system based on text mask collaborative prompting according to claim 7 is characterized in that The semantic segmentation processing module includes: Self-supervised text hint STP branch unit: used to suppress background noise by utilizing cross-layer block association and multi-scale text features; Weakly supervised mask hint WMP branch unit: used to refine boundary accuracy using mask seeds.

Images