Semantic segmentation method and system for open vocabulary image

Abstract

The invention provides an open vocabulary image semantic segmentation method and system, and provides a text guide feature refining module and a structure perception guide enhancement module. Through the method provided by the invention, the semantic correlation region in the visual feature can be selectively focused, and the interference of irrelevant background noise is inhibited, so that the distinguishing capability of categories with similar texture features is effectively improved. Meanwhile, rich space information contained in the CLIP intermediate visual features can be fully mined to guide the up-sampling process of the model, the recovery capability of object structure details is remarkably improved, and the boundary of the object is described more accurately.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to an open-vocabulary image semantic segmentation method and a system thereof. BACKGROUND

[0002] Image semantic segmentation aims to label each pixel of an image with a semantic class, which can be widely applied in embodied intelligence, land surveying and mapping, environmental monitoring, disaster assessment, urban planning, etc. This technology can effectively identify and extract object information, and is an important basis for intelligent interpretation of image data.

[0003] Existing image semantic segmentation methods mostly use deep learning models, such as convolutional neural networks (CNN), encoder-decoder architectures (such as U-Net, DeepLab series), and Transformer-based models. These methods can achieve high segmentation accuracy within a closed class set, but usually require large-scale, class-consistent labeled datasets and assume that the class set of the training set and the test set is the same. However, in practical applications, due to significant differences in image shooting conditions, imaging devices, and observed targets, new classes or rare classes that are not covered in the training set often occur. However, due to the high cost of manual annotation and time consumption, it is difficult to build a training set covering all classes, resulting in a significant decline in model recognition ability in an open environment. To alleviate this problem, few-shot segmentation and zero-shot segmentation methods have been proposed. Few-shot segmentation can adapt to some new classes with limited annotation, but still requires labeled samples of target classes; zero-shot segmentation can eliminate annotation dependence by using external knowledge, but is usually limited to a fixed label space, making it difficult to cope with new classes emerging in real-world application scenarios.

[0004] In recent years, open-vocabulary semantic segmentation (OVSS) methods have received widespread attention. These methods introduce visual-linguistic pre-trained models (such as CLIP) to map image features and text features to a unified feature space, enabling the recognition of classes not present in the training set during inference. However, although these methods have made significant progress in natural image domains, due to the unique characteristics of images in robotics, remote sensing, and other fields in terms of imaging conditions and object distribution, existing open-vocabulary methods often face performance degradation when directly transferred to these scenarios. Therefore, how to balance the accuracy of image segmentation while improving the generalization ability and practicality of the model in an open-class environment has become a core problem that needs to be solved in current technology.

[0005] In practical applications, the following challenges are particularly prominent in robotic and remote sensing image processing: 1) Images often contain semantically different but highly similar texture categories (such as farmland and grassland, rooftops and paved roads), which can easily lead to category confusion. This phenomenon is especially pronounced in weakly supervised open-vocabulary semantic segmentation; 2) In complex scenes, existing methods struggle to accurately depict object boundaries, often resulting in fragmented or blurred outlines in boundary regions, affecting the spatial consistency and semantic coherence of the segmentation results. Current open-vocabulary image semantic segmentation techniques lack effective means to simultaneously address these issues. Summary of the Invention

[0006] This invention is made to solve the above-mentioned problems, and aims to provide an open vocabulary image semantic segmentation method and system.

[0007] This invention provides an open-vocabulary image semantic segmentation method to improve the ability to distinguish texture similarity categories and recover object boundaries in image semantic segmentation. It includes the following steps: S1: Multimodal feature acquisition step, for image... Each pixel in the image is assigned a semantic label. Geometric transformations are performed on the image to enhance it. Then, CLIP's visual encoder and text encoder independently encode the image and its corresponding text category description, respectively, to obtain multimodal feature representations, including visual feature embeddings and text feature embeddings. S2: The visual feature refinement step involves processing the local text feature embeddings into the text-guided feature refinement module TGFRM to obtain semantic attention weights. These weights are then injected into the convolutional block attention module CBAM to refine the visual feature embeddings, enhancing the feature representations relevant to the target semantics while suppressing irrelevant background responses. S3: The feature aggregation step calculates the pixel-level similarity between the refined visual feature embedding and the text feature embedding to generate a preliminary semantic feature map. Then, it is repeatedly refined in two stages to obtain semantic features. S4: Guided feature acquisition step uses the structure-aware guided enhancement module SAGEM to enhance the intermediate features of the CLIP visual encoder, extract spatial information, and obtain guided features. S5: Segmentation result output step activates the guided features along both spatial and channel dimensions. Guided semantic feature upsampling step concatenates the activated guided features and the upsampled semantic features and performs double convolution processing to output the segmentation result.

[0008] The open-vocabulary image semantic segmentation method provided by this invention may also have the following feature: wherein, in S1, the semantic labels come from a set of free-form text descriptions. Each of them Indicates a possible category.

[0009] In the open vocabulary image semantic segmentation method provided by the present application, the visual feature embedding in S1 can be represented as: The text feature embedding can be represented as: , wherein represents the visual encoder of CLIP, represents the text encoder of CLIP, represents the embedding dimension shared by the two modalities, represents the spatial resolution of the visual feature map, represents the number of semantic categories, represents the number of prompt templates used to describe each category.

[0010] In the open vocabulary image semantic segmentation method provided by the present application, the text-guided feature refinement module TGFRM in S2 can have the following features:

[0011] The local text feature embedding is input into a self-attention mechanism to generate a representation with context awareness:

[0012]

[0013] , wherein , a max-pooling operation is applied on the embedding dimension:

[0014]

[0015] , wherein is the intermediate vector after max-pooling, which is then projected into semantic attention weights by a lightweight transformation:

[0016]

[0017] , wherein is implemented by a single-layer perception machine, represents the GELU activation function, represents the Sigmoid function.

[0018] In the open vocabulary image semantic segmentation method provided by the present application, the refined visual feature embedding in S2 can be represented as:

[0019]

[0020] , wherein is the text-guided channel attention map, is the spatial attention map.

[0021] In the open vocabulary image semantic segmentation method provided by the present application, the method can further have the following features: in S3, the step of generating the preliminary semantic feature map is specifically: calculating the pixel-level similarity between the refined visual feature embedding and the text feature embedding through cosine similarity to obtain a similarity map, and then fusing the cross-modal information through a 1x1 convolution operation to generate the preliminary semantic feature map,

[0022] The similarity map is represented as:

[0023] Wherein, represents the similarity map of all categories.

[0024] The preliminary semantic feature map is represented as:

[0025]

[0026] Wherein, represents the rough class-level semantic clue encoding for each spatial position.

[0027] In the open vocabulary image semantic segmentation method provided by the present application, the method can further have the following features: in S3, the two-stage refinement process includes: a first-stage spatial-level aggregation and a second-stage class-level aggregation,

[0028] The spatial-level aggregation is represented as:

[0029]

[0030] Wherein, and respectively represent SwinTransformer modules with local window and shift window attention mechanisms, is the feature map for the mth category,

[0031] The class-level aggregation is represented as:

[0032] Wherein, is a lightweight linear Transformer module, is a pixel point n in the image.

[0033] In the open vocabulary image semantic segmentation method provided by the present application, the method can further have the following features: in S4, the intermediate feature is specifically represented as: ,

[0034] The step of enhancing the intermediate feature of the CLIP visual encoder is specifically:

[0035] Two stacked deformable convolution layers are applied to enhance the adaptability of the intermediate features to irregular object shapes, denoted as:

[0036]

[0037] wherein, a function is used to generate a learnable sampling offset,

[0038] The enhanced intermediate features are input into a mixed upsampling module composed of a bilinear interpolation upsampling and a convolution operation to simultaneously improve the spatial resolution and refine the local feature representation, denoted as:

[0039]

[0040] For the upsampled feature map , a lightweight channel attention mechanism is introduced to capture the global context information at the channel level through global average pooling , followed by two convolutional layers and a nonlinear activation GELU for nonlinear mapping, and finally a Sigmoid function is used to generate normalized weights, denoted as:

[0041]

[0042] wherein, denotes GELU activation, denotes the Sigmoid function, and the obtained weight vector is used for channel-by-channel modulation of each channel,

[0043] The weighted intermediate features are fused with a residual path to output the feature , denoted as:

[0044]

[0045] wherein, is the channel attention weighted feature, and the residual path preserves low-level information.

[0046] In the open vocabulary image semantic segmentation method provided by the present application, the guide feature can also be activated along the spatial and channel dimensions in S5, and the upsampling step of the guide semantic feature is specific steps:

[0047] ​As the guiding feature, denoted as G, statistics are extracted using both global and channel averages, and channel and spatial weights are obtained through convolutional mapping, as follows:

[0048]

[0049] in, ,and and These represent averaging in the spatial and channel dimensions, respectively. After weighted summation, the activated guidance is obtained, which guides the upsampling step of semantic features, and is represented as:

[0050]

[0051] in, This indicates element-wise broadcast multiplication.

[0052] The activated guiding features and upsampled semantic features are concatenated and then processed through a double convolution layer to output the segmentation result, specifically as follows:

[0053] .

[0054] This invention provides an open-vocabulary image semantic segmentation system, characterized by including: a multimodal feature acquisition module for image... Each pixel in the image is assigned a semantic label. Geometric transformations are performed to enhance the image. Then, CLIP's visual encoder and text encoder independently encode the image and its corresponding text category description, respectively, to obtain multimodal feature representations, including visual feature embeddings and text feature embeddings. A visual feature refinement module processes the local text feature embeddings into a text-guided feature refinement module (TGFRM) to obtain semantic attention weights. These weights are then injected into a convolutional block attention module (CBAM) to refine the visual feature embeddings, enhancing features relevant to the target semantics while suppressing irrelevant background responses. A feature aggregation module calculates the pixel-level similarity between the refined visual feature embeddings and the text feature embeddings, generating a preliminary semantic feature map. This map is then repeatedly refined in two stages to obtain semantic features. A guided feature acquisition module uses a structure-aware guided enhancement module (SAGEM) to enhance the intermediate features of the CLIP visual encoder, extracting spatial information to obtain guided features. A segmentation result output module activates the guided features along both spatial and channel dimensions, guiding the semantic feature upsampling step. The activated guided features and upsampled semantic features are concatenated and processed through a double convolutional layer to output the segmentation result.

[0055] The role and effect of invention

[0056] The open vocabulary image semantic segmentation method and system have the following beneficial effects.

[0057] The open vocabulary image semantic segmentation method and system can effectively improve the texture similar class distinguishing ability and object boundary detail recovery ability of the open vocabulary image semantic segmentation system in semantic segmentation, enhance the semantic expression precision and spatial structure perception ability of the model, and further significantly improve the overall segmentation performance and generalization ability.

[0058] For the semantic confusion problem in the image, the application proposes a text-guided feature refinement module (Text-Guided Feature Refinement Module, TGFRM). The module is based on an attention mechanism, and by using class-level text prior information, it guides the adaptive adjustment of channel and spatial attention, effectively suppresses background noise, enhances visual features related to target semantics, and thus improves the distinguishing ability of similar texture classes.

[0059] For the problem of target boundary blur and contour breakage in the image, the application designs a structure perception guided enhancement module (Structure Aware Guidance Enhancement Module, SAGEM). The module fully excavates the rich spatial information contained in the intermediate visual features of CLIP, dynamically adjusts the receptive field and channel attention mechanism, strengthens the geometric structure perception and spatial continuity, guides the upsampling process of the model, improves the detail recovery ability, and realizes more accurate target boundary description. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of the open vocabulary image semantic segmentation method in the embodiment of the application.

[0061] Figure 2 is a flowchart of the processing of the text-guided feature refinement module in the embodiment of the application.

[0062] Figure 3 is a flowchart of the processing of the structure perception guided enhancement module in the embodiment of the application. DETAILED DESCRIPTION

[0063] In order to make the technical means, creative features, purposes and effects achieved by the application easy to understand, the following embodiments will be specifically described in combination with the drawings.

[0064] Figure 1 is a flowchart of the open vocabulary image semantic segmentation method in the embodiment of the application.

[0065] As Figure 1As shown, the open vocabulary image semantic segmentation method of the embodiment is used to improve the distinguishing ability of texture similar categories and the recovery ability of object boundaries in image semantic segmentation, and includes the following steps:

[0066] Step S1 is a multi-modal feature acquisition step. A semantic label is assigned to each pixel in the image . The semantic label comes from a set of free-form text descriptions , where each represents a possible category expressed in natural language.

[0067] The image is geometrically transformed for enhancement, and then the visual encoder and the text encoder of CLIP are used to independently encode the image and the text category description corresponding to the image, respectively, to obtain multi-modal feature representations, including visual feature embeddings and text feature embeddings.

[0068] The visual feature embedding is represented as: , and the text feature embedding is represented as: , where represents the visual encoder of CLIP, represents the text encoder of CLIP, represents the embedding dimension shared by the two modalities, represents the spatial resolution of the visual feature map, represents the number of semantic categories, represents the number of prompt templates used to describe each category.

[0069] Step S2 is a visual feature refinement step. The local text feature embedding is input into the text-guided feature refinement module TGFRM for processing to obtain semantic attention weights. The semantic attention weights are injected into the convolution block attention module CBAM to refine the visual feature embedding, enhance the feature expression related to the target semantics, and suppress irrelevant background responses.

[0070] Figure 2 is a flowchart of the processing of the text-guided feature refinement module of the embodiment of the application;

[0071] As Figure 2 shown, in order to obtain semantic priors for the guided refinement process, a set of text-driven attention weights is calculated. The local text feature embedding is input into a self-attention mechanism to generate a representation with context awareness:

[0072]

[0073] , where In order to further reduce redundancy and retain the most discriminative signals in each feature channel, a max-pooling operation is applied in the embedding dimension:

[0074]

[0075] wherein, is the max-pooled intermediate vector, which is then projected into semantic attention weights by a light-weight transformation :

[0076]

[0077] wherein, is realized by a single-layer perceptron, denotes the GELU activation function, denotes the Sigmoid function. The processed semantic weights reflect the importance of each class and are used to guide the subsequent visual refinement process.

[0078] Based on the convolutional block attention module CBAM, which also combines channel attention and spatial attention to adaptively recalibrate visual features. Specifically, the semantic attention weights are injected into the convolutional block attention module CBAM to guide the generation of cross-modal attention maps. The final refined visual feature embedding is denoted as:

[0079]

[0080] wherein, is the text-guided channel attention map, is the spatial attention map. The introduction of this design before similarity calculation adjusts the visual feature embedding, which can help the model focus on semantically relevant areas, enhance the expression of features related to the target semantics, and suppress irrelevant background noise, increasing the discriminability of the calculated similarity and improving the cross-modal correspondence ability, thereby alleviating potential semantic confusion when dealing with features with similar texture patterns.

[0081] Step S3 is a feature aggregation step. The pixel-level similarity between the refined visual feature embedding and the text feature embedding is calculated to generate a preliminary semantic feature map, which is then repeatedly refined by two-stage refinement processing to obtain the semantic feature.

[0082] The pixel-level similarity between the refined visual feature embedding and the text feature embedding is calculated by cosine similarity to measure the alignment degree between the visual and text modalities, and the similarity map is obtained, denoted as:

[0083] wherein, denotes the similarity map of all classes.

[0084] Then the similarity map is fused with cross-modal information through a 1x1 convolution operation to generate a preliminary semantic feature map, denoted as:

[0085]

[0086] wherein, represents the coarse class-level semantic cue encoding for each spatial position.

[0087] To enhance the expression ability of semantic features, a two-stage refinement process is adopted.

[0088] The first stage performs spatial aggregation to improve long-distance context reasoning ability, denoted as:

[0089]

[0090] wherein, and respectively represent SwinTransformer modules with local window and shift window attention mechanisms, is the feature map for the m-th class. The role of this operation is that for each class m, the model will take its corresponding feature map and then perform spatial aggregation through two transformer modules, thereby suppressing background noise and making the similarity information of pixel regions belonging to the same class more accurate and coherent.

[0091] The second stage adopts class-level aggregation to model the dependency between classes:

[0092]

[0093] wherein, is a lightweight linear Transformer module, is a pixel point n in the image. That is, for each pixel point n, its feature map is taken. Through linear Transformer modeling of the relationship between different classes, information flow within the label space is promoted, thereby enhancing the generalization ability of the model to new classes by utilizing the semantic association between labels. Through repeated alternating application of spatial-level aggregation and class-level aggregation, these rough semantic feature maps are gradually refined, ultimately achieving more accurate and discriminative segmentation results, and obtaining more accurate semantic features.

[0094] S4: Guided feature acquisition step, using the structure perception guided enhancement module SAGEM to enhance the intermediate features of the CLIP visual encoder, extract spatial information, and obtain guided features.

[0095] Figure 3is a flowchart of the process handled by the structure-aware guidance enhancement module in the embodiment of the present application.

[0096] As shown in Figure 3 , for an intermediate feature of the CLIP visual encoder, two stacked deformable convolution layers are first applied to improve the adaptability to irregular object shapes, denoted as:

[0097]

[0098] wherein, the function is used to generate a learnable sampling offset, so that the filter can better capture structural changes.

[0099] The enhanced intermediate feature is first passed through a hybrid upsampling module, which is composed of a bilinear interpolation upsampling and a convolution operation to simultaneously improve the spatial resolution and refine the local feature representation, denoted as:

[0100]

[0101] For the upsampled feature map , a lightweight channel attention mechanism is introduced to capture global context information at the channel level through global average pooling , followed by two convolutional layers and a GELU non-linear activation for non-linear mapping, and finally a Sigmoid function is used to generate normalized weights, denoted as:

[0102]

[0103] wherein, represents the GELU activation, represents the Sigmoid function, and the obtained weight vector is used to modulate each channel channel-by-channel, thereby adaptively highlighting key features and suppressing redundant information.

[0104] At the same time, a residual path shortcut is designed in the branch, which includes convolution operation (channel alignment), bilinear interpolation (spatial alignment), normalization and non-linear activation, so as to ensure the consistency of the input and output in dimension, and alleviate the information loss caused by deep upsampling.

[0105] Finally, the weighted intermediate feature is fused with a residual path to output the feature , denoted as:

[0106]

[0107] in, These are features after channel attention weighting. It is the low-level information preserved by the residual path. The combination of the two can take into account both fine-grained spatial details and robustness of global context.

[0108] The residual path here is achieved through... The same interpolation and convolution are used directly for upsampling. Through structural modeling, attention optimization, and residual preservation, SAGEM enhances intermediate features, providing stronger upsampling-guided features that enable the model to better recover spatial details and accurately depict object boundaries in complex scenes.

[0109] Step S5 is the segmentation result output step. The guiding features are activated along both spatial and channel dimensions, and the guiding semantic features are upsampled. The activated guiding features and the upsampled semantic features are concatenated and processed through a double convolution layer to output the segmentation result.

[0110] Will As the guiding feature, denoted as G, statistics are extracted using both global and channel averages, and channel and spatial weights are obtained through convolutional mapping, as follows:

[0111]

[0112] in, ,and and These represent averaging in the spatial and channel dimensions, respectively. After a simple weighted summation, the activated guidance is obtained, which guides the upsampling step of semantic features, represented as:

[0113]

[0114] in, This indicates element-wise broadcast multiplication.

[0115] The activated guiding features and upsampled semantic features are concatenated and then processed through a double convolution layer to output the segmentation result, specifically as follows:

[0116] .

[0117] This embodiment also provides an open-vocabulary image semantic segmentation method and system, including:

[0118] The multi-modal feature acquisition module is configured to implement step S1, i.e., assigning a semantic label to each pixel in the image I. The image is subjected to geometric transformation enhancement, and then the visual encoder and the text encoder of the CLIP are used to independently encode the image and the text category description corresponding to the image, respectively, to obtain multi-modal feature representations, including visual feature embeddings and text feature embeddings.

[0119] The visual feature refinement module is configured to implement step S2, i.e., inputting the local text feature embeddings into the text-guided feature refinement module TGFRM for processing to obtain semantic attention weights. The semantic attention weights are injected into the convolution block attention module CBAM to refine the visual feature embeddings, thereby enhancing the feature expression related to the target semantics and suppressing irrelevant background responses.

[0120] The feature aggregation module is configured to implement step S3, i.e., calculating the pixel-level similarity between the refined visual feature embeddings and the text feature embeddings to generate a preliminary semantic feature map, and then repeatedly performing two-stage refinement processing on the preliminary semantic feature map to obtain semantic features.

[0121] The guided feature acquisition module is configured to implement step S4, i.e., using the structure-aware guided enhancement module SAGEM to enhance the intermediate features of the CLIP visual encoder to extract spatial information and obtain guided features.

[0122] The segmentation result output module is configured to implement step S5, i.e., activating the guided features along the spatial and channel dimensions, guiding the upsampling step of the semantic features, concatenating the activated guided features and the upsampled semantic features, and performing double-layer convolution processing to output the segmentation result.

[0123] Effects of the embodiment

[0124] According to the open-vocabulary image semantic segmentation method and system, the following beneficial effects are achieved:

[0125] The open-vocabulary image semantic segmentation system can effectively improve the ability to distinguish similar texture classes and the ability to recover object boundary details in semantic segmentation, enhance the precision of semantic expression and the ability of spatial structure perception of the model, and thus significantly improve the overall segmentation performance and generalization ability. Specifically:

[0126] To address the semantic confusion problem in images, the present application proposes a text-guided feature refinement module (TGFRM). Based on the attention mechanism, the module uses class-level text prior information to guide the adaptive adjustment of channel and spatial attention, effectively suppresses background noise, and enhances visual features related to target semantics, thereby improving the ability to distinguish similar texture classes.

[0127] In view of the problems of target boundary blur and contour breakage in the image, a structure perception guidance enhancement module (SAGEM) is designed.

[0128] Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application, and various changes and improvements can be made without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. An open-vocabulary image semantic segmentation method, used to improve the ability to distinguish texture similarity categories and recover object boundaries in image semantic segmentation, characterized in that, Includes the following steps: S1: Multimodal feature acquisition step, for image Each pixel in the image is assigned a semantic label. The image is then geometrically transformed and enhanced. Finally, CLIP's visual encoder and text encoder are used to independently encode the image and its corresponding text category description, respectively, to obtain a multimodal feature representation, including visual feature embedding and text feature embedding. S2: Visual feature refinement step, embed the local text features into the input text guided feature refinement module TGFRM for processing to obtain semantic attention weights, inject the semantic attention weights into the convolutional block attention module CBAM to refine the visual feature embedding, enhance the feature expression related to the target semantics, and suppress irrelevant background responses; S3: Feature aggregation step, calculate the pixel-level similarity between the refined visual feature embedding and the text feature embedding, generate a preliminary semantic feature map, and then repeatedly perform two-stage refinement processing on it to obtain semantic features; S4: Guided feature acquisition step, using the structure-aware guided enhancement module SAGEM to enhance the intermediate features of the CLIP visual encoder, extract spatial information, and obtain guided features; S5: Segmentation result output step, the guiding feature is activated along the spatial and channel dimensions, the semantic feature is upsampled, the activated guiding feature and the upsampled semantic feature are concatenated and processed by double convolution, and the segmentation result is output.

2. The open-vocabulary image semantic segmentation method according to claim 1, characterized in that: in, In S1, the semantic tags come from a set of free-form text descriptions. Each of them Indicates a possible category.

3. The open-vocabulary image semantic segmentation method according to claim 2, characterized in that: in, In S1, the visual feature embedding is represented as: The text feature embedding is represented as: ,in, This refers to the visual encoder of CLIP. The text encoder representing CLIP. This represents the embedding dimension shared by the two modalities. This represents the spatial resolution of the visual feature map. Indicates the number of semantic categories, This indicates the number of prompt templates used to describe each category.

4. The open-vocabulary image semantic segmentation method according to claim 3, characterized in that: in, In step S2, the specific steps of the text-guided feature refinement module TGFRM in processing the local text feature embedding are as follows: Embedding local text features The input is fed into a self-attention mechanism to generate a context-aware representation: ， in, Apply max pooling to the embedding dimension: ， in, The intermediate vector after max pooling is then projected into semantic attention weights through a lightweight transformation. : ， in, Implemented by a single-layer perceptron. This represents the GELU activation function. This represents the Sigmoid function.

5. The open-vocabulary image semantic segmentation method according to claim 4, characterized in that: in, In S2, the refined visual feature embedding Represented as: ， in, It is a text-guided channel attention map. It is a spatial attention map.

6. The open-vocabulary image semantic segmentation method according to claim 5, characterized in that: in, In step S3, the preliminary semantic feature map generation process specifically involves: calculating the pixel-level similarity between the refined visual feature embedding and the text feature embedding using cosine similarity to obtain a similarity map; then, fusing cross-modal information through a 1×1 convolution operation to generate the preliminary semantic feature map. The similarity graph is represented as follows: , in, A similarity graph representing all categories; The preliminary semantic feature map is represented as follows: ， in, This represents a coarse-level semantic cue encoding for each spatial location.

7. The open-vocabulary image semantic segmentation method according to claim 6, characterized in that: in, In S3, the two-stage refinement process includes: spatial-level aggregation in the first stage and category-level aggregation in the second stage. Spatial-level aggregation is represented as: ， in, and These represent the Swing Transformer modules with local window and shift window attention mechanisms, respectively. For the feature map of the m-th category, Category-level aggregation is represented as: , in, It is a lightweight linear Transformer module. Let n be a pixel in the image.

8. The open-vocabulary image semantic segmentation method according to claim 1, characterized in that: in, In S4, the intermediate features are specifically represented as follows: , The specific steps for enhancing the intermediate features of the CLIP visual encoder are as follows: By applying two stacked deformable convolution layers, the adaptability of intermediate features to irregular object shapes is enhanced, as shown below: ， in, The function is used to generate learnable sampling offsets. Enhanced intermediate features After passing through the hybrid upsampling module, the hybrid upsampling module consists of bilinear interpolation upsampling. With a convolution operation Composition, used to simultaneously improve spatial resolution and refine local feature representation, is expressed as: ， For the upsampled feature map A lightweight channel attention mechanism is introduced, which utilizes global average pooling. The system captures global context information at the channel level, then performs non-linear mapping using two convolutional layers and the non-linear activation GELU, and finally generates normalized weights using the Sigmoid function, as follows: ， in, This indicates that GELU is activated. The weight vector obtained by representing the Sigmoid function. Used for channel-by-channel modulation of each channel. The weighted intermediate features are fused with a residual path to output the features. , is represented as: ， in, These are features after channel attention weighting. It is the low-level information preserved by the residual path.

9. The open-vocabulary image semantic segmentation method according to claim 8, characterized in that: in, In step S5, the guiding features are activated along both spatial and channel dimensions, guiding the upsampling step of semantic features. The specific steps are as follows: Will As the guiding feature, denoted as G, statistics are extracted using both global and channel averages, and channel and spatial weights are obtained through convolutional mapping, as follows: ， in, ,and and These represent averaging in the spatial and channel dimensions, respectively. After weighted summation, the activated guidance is obtained, which guides the upsampling step of the semantic features, and is expressed as: ， in, This indicates element-wise broadcast multiplication. The activated guiding features and the upsampled semantic features are concatenated and then processed through a double convolutional layer to output the segmentation result, specifically as follows: 。 10. An open-vocabulary image semantic segmentation system, characterized in that, include: The multimodal feature acquisition module is used for image processing. Each pixel in the image is assigned a semantic label. The image is then geometrically transformed and enhanced. Finally, CLIP's visual encoder and text encoder are used to independently encode the image and its corresponding text category description, respectively, to obtain a multimodal feature representation, including visual feature embedding and text feature embedding. The visual feature refinement module embeds the local text features into the input text guided by the feature refinement module TGFRM for processing to obtain semantic attention weights. The semantic attention weights are then injected into the convolutional block attention module CBAM to refine the visual feature embedding, enhance the feature expression related to the target semantics, and suppress irrelevant background responses. The feature aggregation module calculates the pixel-level similarity between the refined visual feature embedding and the text feature embedding, generates a preliminary semantic feature map, and then repeatedly performs a two-stage refinement process on it to obtain semantic features. The guided feature acquisition module uses the structure-aware guided enhancement module SAGEM to enhance the intermediate features of the CLIP visual encoder, extract spatial information, and obtain guided features. The segmentation result output module activates the guiding features along both spatial and channel dimensions, guides the upsampling step of the semantic features, concatenates the activated guiding features and the upsampled semantic features, and performs double convolution processing to output the segmentation result.

Images