A single-stage zero-shot semantic segmentation method and device based on CLIP

Abstract

The present application provides a single-stage zero-shot semantic segmentation method and device based on CLIP, which belongs to the technical field related to computer vision and deep learning. A feature self-calibration module is introduced to improve the model's ability to focus on local details of foreground objects by recalibrating the features of the image encoder, thereby solving the problem of local detail loss and enabling the model to better handle refined tasks; an independent text-image decoder is designed to align the visual features of the intermediate layer with the text embedding, thereby making full use of the rich features of the intermediate layer and improving the expressiveness of the model; a cascaded feature fusion method is adopted to achieve effective fusion of the visual features of the intermediate layer and the last layer, thereby avoiding the problem of feature correlation being destroyed and significantly improving the cross-modal alignment capability of the model; overfitting is alleviated by deep cross-visual cue adjustment, and a cross-attention mechanism is used to promote the interaction between pre-trained tags and cue tags to enhance the adaptability to unseen categories.

Description

Technical Field

[0001] The present application relates to the technical fields related to computer vision and deep learning, and in particular to a single-stage zero-sample semantic segmentation method and device based on CLIP. Background Art

[0002] Semantic segmentation is an important task in computer vision, which aims to classify each pixel in an image and mark the regions belonging to different categories. In the past few years, with the advancement of deep learning technology, especially the application of convolutional neural network (CNN) and Transformer architecture, semantic segmentation technology has made significant progress. Traditional semantic segmentation methods mostly rely on large-scale annotated datasets and can only segment fixed categories defined in the training phase. In recent years, the research on zero-shot semantic segmentation (ZSS) has gradually attracted attention. The goal of zero-shot semantic segmentation is to enable the model to not only segment the categories that appear in the training set, but also to handle new categories that do not exist in the training set. The development of zero-shot semantic segmentation technology is closely related to visual language models such as CLIP. CLIP can recognize new categories of images without specific category labels by jointly training images and text, thus providing an effective solution for zero-shot tasks.

[0003] Visual language models aim to understand the complex connections between visual elements and their text. A key example of such models is CLIP, which uses a large dataset of 400 million Internet-sourced image-text pairs to connect language to images. CLIP uses contrastive learning to align images and text in a shared feature space. This capability enables CLIP to take on a variety of computer vision tasks and extend to other domains. CLIP's ability to apply its knowledge to a wide range of domains and formats highlights its strengths, especially in performing zero-shot tasks.

[0004] Since visual language models (such as CLIP) mainly generate image-level features, and the semantic segmentation task requires the model to output pixel-level segmentation masks, it is not enough to directly apply visual language models for semantic segmentation. Cascaded text-image decoders achieve pixel-level segmentation by gradually processing the output features of the visual language model using multiple decoders. These decoders are able to gradually refine image features to improve segmentation accuracy, avoiding the shortcomings of traditional methods that directly rely on image-level features.

[0005] In the prior art, CLIP-based methods usually only use the visual features of the last layer of the image encoder to align text embeddings, while ignoring the rich object detail information in the middle layer. This single-level feature alignment method will lead to low feature utilization and fail to fully utilize the potential of the image encoder, especially in cross-modal tasks that require fine-grained information. Although some methods attempt to fuse the intermediate layer features with the last layer of visual features to enrich feature expression, this approach will destroy the correlation between visual features and text embeddings in actual tasks. Since multi-layer features may have expression differences or noise, the fusion process increases the complexity of the model for the visual language alignment task, and at the same time increases the difficulty of aligning visual features with text embeddings during fine-tuning. .

[0006] In tasks with unseen categories, the method of fusing the features of the middle layer and the last layer not only fails to improve the generalization ability of the model, but may also cause performance degradation due to the problem of feature correlation destruction mentioned above. This is reflected in the fact that the model's performance on unknown categories is significantly worse than that on known categories, limiting its scalability in practical applications.

[0007] The CLIP method relies on a single decoder (text-image decoder) when aligning the last layer of visual features with text embeddings, which fails to effectively control the model complexity and easily leads to overfitting. In particular, in scenarios with limited or uneven data distribution, the model's dependence on training data is further enhanced, weakening its generalization ability in diverse tasks. Summary of the invention

[0008] The embodiment of the present application provides a single-stage zero-shot semantic segmentation method and device based on CLIP, which aims to alleviate the overfitting problem through deep cross visual cue adjustment (DCVPT), ​​and use the cross-attention mechanism to promote the interaction between pre-trained tags and cue tags to enhance the adaptability to unseen categories; combine the cascade decoder framework to align visual features and text embeddings in stages, generate multi-level segmentation masks, and avoid feature fusion conflicts; optimize the attention area through the feature self-calibration module (FCM) to extract global background and local detail information; introduce a dual-path decoder in the final stage to capture global semantic associations through bidirectional queries, significantly improving the segmentation accuracy in complex scenes and the generalization ability of unseen categories.

[0009] In a first aspect, the present application provides a single-stage zero-shot semantic segmentation method based on CLIP for computer vision processing, wherein the single-stage zero-shot semantic segmentation method based on CLIP comprises:

[0010] S1. The input image is divided into patches of fixed size and fed into the pre-trained CLIP image encoder. Through the deep cross-visual cue adjustment module, the cross-attention of patch labels and cue labels is calculated, and the results are integrated into the visual features to generate enhanced features adapted to unseen categories.

[0011] S2. Combine the category embedding generated by the text encoder with the global features generated by the image encoder and categorical features The two components are sent together to the relationship description module, which generates multi-level classification features through multi-level feature alignment, and then input into the decoder of the corresponding stage to complete the alignment and enhancement of visual-text features.

[0012] S3, introduce a feature self-calibration module and a neighborhood Gaussian aggregation module in each decoder to optimize global background extraction and local feature calibration respectively, while enhancing the focus on foreground targets and reducing the interference of noise on feature extraction;

[0013] S4, using a cascaded decoder framework to perform staged feature extraction on specific blocks output by the image encoder, using an independent decoder at each stage to align visual features with text embeddings, and gradually generate multi-level segmentation masks through feature fusion;

[0014] S5. Integrate the segmentation masks generated layer by layer to form a final segmentation result, wherein the output segmentation result includes the foreground object category and its pixel-level position distribution.

[0015] Optionally, the S1 specifically includes:

[0016] The input image sample is first divided into fixed-size image patches, each patch contains a fixed pixel block, and the divided image patches are sent to the pre-trained CLIP image encoder to extract patch marker features through preliminary processing of the encoder. ;

[0017] At each layer of the CLIP image encoder, the patch marker features are first calculated Self-attention to capture correlations between patches within the same image;

[0018] Mark the text prompt As auxiliary information, and the enhanced patch features Calculate cross attention together to further integrate the information between vision and text;

[0019] After the cross attention calculation is completed, the fused features are integrated back into the original features through residual connections to ensure that the basic information is retained while enhancing the features. Then, the high-level features are further extracted through the multi-layer perceptron MLP: , where MLP is a multi-layer perceptron. For cross attention, For the visual features after processing;

[0020] The processed visual features As the input of the subsequent decoder module, it is used to generate multi-level segmentation masks.

[0021] Optionally, the S2 specifically includes:

[0022] The text encoder generates category embedding based on the input prompt text. The category embedding contains the semantic information of the category. At the same time, the image encoder generates global features and classification features after processing the input image. Among them, the global features reflect the overall semantic information of the entire image, and the classification features contain the detailed details of the local area of ​​the image.

[0023] The generated category embedding and image features are fed into the relationship description module, which achieves deep fusion of visual and textual information through a multi-level alignment strategy;

[0024] The enhanced features after multi-level alignment are passed to the cascaded decoder framework in sequence according to a staged design. In each stage, the decoder further utilizes the enhanced features after multi-level alignment to refine the alignment of visual and textual features, and gradually generates multi-level segmentation masks to provide layer-by-layer optimization for the final segmentation results.

[0025] Optionally, the S3 specifically includes:

[0026] In each decoder, the input features are first subjected to neighborhood Gaussian aggregation. The neighborhood Gaussian aggregation module uses Gaussian distribution weights to perform weighted fusion on the features in the neighborhood with a specific pixel as the center to generate local aggregated features.

[0027] Based on the features processed by the neighborhood Gaussian aggregation module, the global features are further adjusted and optimized through the feature self-calibration module. The feature self-calibration module learns the relationship between different areas in the feature map through the global self-attention mechanism, generates a global calibration weight matrix, and uses it to re-weight the input feature map.

[0028] Optionally, the feature self-calibration module is further used to adjust and optimize the global feature on the feature after the operation and processing of the neighborhood Gaussian aggregation module. The feature self-calibration module learns the relationship between different regions in the feature map through the global self-attention mechanism, generates a global calibration weight matrix, and re-weights the input feature map based on it, specifically including:

[0029] On the features processed by the neighborhood Gaussian aggregation modeling operation, the input feature map first undergoes global pooling operations in two directions: horizontal pooling and vertical pooling. The global context information is extracted horizontally and vertically respectively to generate two feature vectors. and , and Represent the global feature distribution of the input feature map in the horizontal and vertical directions respectively. Then, the feature vector is broadcasted and Perform element-by-element addition to fuse information from two directions and generate a globally focused weighted feature map.

[0030] After completing the global feature fusion, the generated weighted feature map is subjected to strip convolution in two directions. Convolution adjusts each row of features to better fit the horizontal shape of the foreground object, and uses Convolution calibrates each column of features to fit the vertical shape of the foreground object. After convolution, the weighted features are normalized by batch normalization, and a nonlinear transformation is introduced through the ReLU activation function to further enhance the feature response to the foreground area.

[0031] After completing the calibration of the strip convolution, the feature map undergoes a 3×3 depthwise convolution operation to further extract local detail information. The globally calibrated features are further refined for the edges and small-scale areas in the feature map. Then, the calibrated global attention features are fused with the convolved input feature map through the Hadamard product to weight the calibration information into the original input features.

[0032] To further improve the model's ability to reuse features, the fused features are added to the original input feature map through residual connections to generate the final output feature map.

[0033] Optionally, the S4 specifically includes:

[0034] A cascaded decoder framework is used to perform staged feature extraction on specific blocks output by the image encoder. In the first stage, the image features are first aggregated by the neighborhood Gaussian aggregation module to aggregate the salient features of the local spatial neighborhood, thereby capturing the spatial relationship of the key area. Subsequently, the image features are self-calibrated by the feature self-calibration module to further enhance the semantic consistency of the features. Then, the image features are aligned with the text embedding through the cross-modal attention mechanism to generate a preliminary segmentation mask. , realize the fusion and semantic extraction of multimodal information;

[0035] In the second stage, the input image feature blocks are once again processed by the feature self-calibration module and the neighborhood Gaussian aggregation module to enhance their semantic expression ability and consistency of spatial representation. On this basis, the semantic details are optimized to generate a more refined segmentation mask. , improving the model's ability to depict boundaries and semantically complex areas.

[0036] In the third stage, the input image feature blocks are further combined with global and local multimodal information using the enhanced feature self-calibration module and the dual-path decoder. The dual-path decoder deeply fuses the image features and text embeddings to generate the final segmentation mask. .

[0037] Optionally, the cascade decoder framework is used to perform staged feature extraction on specific blocks output by the image encoder. In the first stage, the image features are first aggregated by the neighborhood Gaussian aggregation module to aggregate the salient features of the local spatial neighborhood, thereby capturing the spatial relationship of the key area. Subsequently, the image features are self-calibrated by the feature self-calibration module to further enhance the semantic consistency of the features. Then, the image features are aligned with the text embedding through the cross-modal attention mechanism to generate a preliminary segmentation mask. , realizing the fusion and semantic extraction of multimodal information, including:

[0038] The image features and text embedding The image features V processed by the feature self-calibration module and the neighborhood Gaussian aggregation module are input into the text-image decoder, and the input features are standardized by the normalization operation;

[0039] The normalized text features and image features Input into the relationship description module to generate relationship-enhanced text features , for enhanced text features Map to the new representation space through linear transformation;

[0040] Image features Through two independent linear layers, the key K and value V are generated respectively. With key Perform matrix multiplication to generate an attention score matrix and scale the attention scores to reduce the sensitivity of high-dimensional features to attention;

[0041] Perform matrix weighting on the attention score matrix and value V after scaling to obtain the output feature after weighted summation , after obtaining the weighted sum of the output features Finally, the model introduces residual connections and layer normalization operations.

[0042] Optionally, in the third stage, the input image feature block is further combined with global and local multimodal information using the enhanced feature self-calibration module and the dual-path decoder, and the dual-path decoder deeply fuses the image features and text embedding to generate the final segmentation mask. , including:

[0043] Accept the text feature T and the image feature g as well as the image feature V processed by the feature self-calibration module and the neighborhood Gaussian aggregation module into the dual-path decoder, and use the normalization operation to standardize the input features respectively;

[0044] The normalized text features and image features Input into the relationship description module to generate relationship-enhanced text features , and then the text features Perform multi-head attention operations, then perform residual connection and normalization operations to obtain the normalized and residual connected output ;

[0045] Normalized and residual connected output And the image features processed by the feature self-calibration module and the neighborhood Gaussian aggregation module As the input of the next multi-head attention, it is then normalized and residually connected to obtain the output of the normalization and residual connection. ;

[0046] The output of the normalized and residual connections Input the multi-layer perceptron module and then go through normalization and residual connection operations to get the normalized and residual connected output ;

[0047] The output of the normalized and residual connections Image features processed by the feature self-calibration module and the neighborhood Gaussian aggregation module As the input of the next multi-head attention, it is then normalized and residually connected to obtain the output of the normalization and residual connection. ;

[0048] The output of the normalized and residual connections Input the multi-layer perceptron module and then go through normalization and residual connection operations to get the normalized and residual connected output ;

[0049] The output of the normalized and residual connections and Perform matrix multiplication.

[0050] Optionally, the S5 specifically includes:

[0051] Through the layer-by-layer decoder , combining visual features and text embedding to generate segmentation masks at each level. The segmentation masks at each level represent the predictions of different feature levels on the segmentation area, capturing feature information from low-level to high-level;

[0052] The segmentation masks generated layer by layer are integrated through weighted summation, and the results are normalized into the final segmentation prediction using the Softmax function;

[0053] The difference between the integrated segmentation mask and the true label is calculated through the pixel-level loss function Optimize to form the final segmentation result, where the pixel-level loss function It is composed of the Dice Loss function and the Focal Loss function, which enhances the overlap measurement of the foreground area and the robustness to unbalanced samples. The final segmentation result is a probability distribution matrix that contains the target foreground object category and its pixel-level position distribution.

[0054] In a second aspect, the present application also provides a CLIP-based single-stage zero-shot semantic segmentation device, wherein the CLIP-based single-stage zero-shot semantic segmentation device is configured to execute the above-mentioned CLIP-based single-stage zero-shot semantic segmentation method.

[0055] Beneficial technical effects of this application:

[0056] The present application provides a single-stage zero-shot semantic segmentation method and device based on CLIP, which introduces a feature self-calibration module, and improves the model's ability to focus on local details of foreground objects by recalibrating the features of the image encoder, thereby solving the problem of local detail loss and enabling the model to better handle refined tasks; a multi-level feature alignment scheme is proposed, and an independent text-image decoder is designed to align the visual features of the middle layer with the text embedding, thereby making full use of the rich features of the middle layer and improving the expressiveness of the model; a cascaded feature fusion method is adopted to achieve effective fusion of the visual features of the middle layer and the last layer, avoiding the problem of feature correlation being destroyed and significantly improving the cross-modal alignment capability of the model; overfitting problems are alleviated through deep cross-visual cue adjustment, and the cross-attention mechanism is used to promote the interaction between pre-trained tags and cue tags to enhance the adaptability to unseen categories; the feature self-calibration module is used to optimize the attention area and extract global background and local detail information; a dual-path decoder is introduced in the final stage to capture global semantic associations through bidirectional queries, significantly improving the segmentation accuracy in complex scenes and the generalization capability of unseen categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 A schematic diagram of a denoising strategy for a consistency model in a MOT task provided in this application;

[0059] Figure 2 Schematic diagram of a proposed consistency track training procedure provided for this application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0061] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0062] It should be understood that in the present application, "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0063] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0064] refer to Figure 1 As shown, an embodiment of the present invention provides a single-stage zero-sample semantic segmentation method based on CLIP for computer vision processing. The single-stage zero-sample semantic segmentation method based on CLIP includes:

[0065] S1. The input image is divided into patches of fixed size and fed into the pre-trained CLIP image encoder. Through the deep cross-visual cue adjustment module, the cross-attention of patch labels and cue labels is calculated, and the results are integrated into the visual features to generate enhanced features adapted to unseen categories.

[0066] Specifically, the input image sample is first divided into fixed-size image patches, each of which contains a fixed pixel block. This segmentation method can ensure that the model processes visual information at a uniform granularity. The divided image patches are sent to the pre-trained CLIP image encoder, and the patch marking features are extracted through preliminary processing by the encoder. , patch marking features As a low-level representation of visual information, it is used for subsequent enhancement calculations;

[0067] At each layer of the CLIP image encoder, the patch marker features are first calculated To capture the correlation between patches in the same image, the correlation between patch features is enhanced by calculating self-attention. Specifically, the formula To realize the calculation of self-attention, represents the patch features enhanced by the self-attention mechanism, The representation represents the self-attention mechanism, which is used to calculate the global correlation between different patch features; It is the first layer normalization operation, which is used to stabilize the feature distribution and alleviate the gradient disappearance problem. It is a feature projection function that maps patch features to query, key, and value spaces for calculating attention scores. Through the self-attention mechanism, the interactivity between enhanced patch features is enhanced, enabling the model to extract richer local and global visual information.

[0068] Mark the text prompt As auxiliary information, and the enhanced patch features Calculate the cross attention together to further integrate the information between vision and text. The specific process can be expressed by formula To achieve this, For cross-attention. is a cross-attention mechanism used to embed textual prompts into Semantic information in and enhanced patch features Combined. Represents a query extracted from visual features. Represents the key and value extracted from the text prompt. Through cross-attention, the model can better understand the semantic meaning of the prompt information and map these semantics to the corresponding visual areas, thereby improving the ability to adapt to unseen categories.

[0069] After the cross attention calculation is completed, the fused features are integrated back into the original features through residual connections to ensure that the basic information is retained while enhancing the features. Then, the high-level features are further extracted through the multi-layer perceptron MLP: ,in, For cross attention, To process the visual features; MLP is a multi-layer perceptron, which is a combination of a set of fully connected layers and nonlinear activation functions, used to mine deep information in the fusion features. Through residual connections, information loss is avoided and the training convergence of the model is accelerated.

[0070] The processed visual features As the input of the subsequent decoder module, it is used to generate multi-level segmentation masks. , which combines the global correlation information of patch features and the semantic information of cue tags. These enhanced features not only contain the high-resolution visual features of the input image, but also improve the generalization ability of unseen categories through the role of cue tags.

[0071] S2. Combine the category embedding generated by the text encoder with the global features generated by the image encoder and categorical features The relationship description module generates multi-level classification features through multi-level feature alignment. , and further input into the decoder of the corresponding stage to complete the alignment and enhancement of visual-text features.

[0072] Class Embeddings generated by the text encoder and global features generated by the image encoder and categorical features The relationship description module generates , and further input into the decoder of the corresponding stage to complete the alignment and enhancement of visual-text features. The specific implementation process is as follows:

[0073] The text encoder generates class embeddings based on the input prompt text. Class embeddings contain the semantic information of the category, which serves as the core basis for subsequent alignment with visual features. At the same time, the image encoder generates global features and classification features after processing the input image. The global features reflect the overall semantic information of the entire image, and the classification features contain the detailed details of the local area of ​​the image. The generation of global features and classification features together lays the foundation for the multimodal fusion of segmentation tasks.

[0074] The generated category embedding and image features are fed into the relational description module, which achieves deep fusion of visual and textual information through a multi-level alignment strategy. Specifically, the relational description module uses the attention mechanism to calculate the correlation between category embedding and image features, embeds category information into image features, emphasizes the similarity between similar features, and suppresses the interference of irrelevant features. After this feature alignment process, the output enhanced features fuse category semantic information and visual details, providing strong contextual support for segmentation tasks.

[0075] The enhanced features after multi-level alignment are sequentially passed to the cascade decoder framework according to the staged design. In each stage, the decoder further uses the enhanced features after multi-level alignment to fine-tune the alignment of visual and text features, gradually generates multi-level segmentation masks, and provides layer-by-layer optimization for the final segmentation results. Through this gradually refined decoding process, the model can better handle complex scenes and improve the segmentation accuracy of fine-grained areas.

[0076] S3. Introduce a feature self-calibration module and a neighborhood Gaussian aggregation module in each decoder to optimize global background extraction and local feature calibration respectively, thereby enhancing the focus on foreground targets while reducing the interference of noise on feature extraction.

[0077] In each decoder, the input features are first subjected to neighborhood Gaussian aggregation. The neighborhood Gaussian aggregation module (NGA) takes a specific pixel as the center and uses Gaussian distribution weights to perform weighted fusion of features in the neighborhood to generate local aggregate features. Specifically, in each decoder, the input features are first subjected to neighborhood Gaussian aggregation. The core goal of the neighborhood Gaussian aggregation module (NGA) is to perform weighted aggregation and refinement of feature information in a local range based on the Gaussian weighting mechanism of the pixel neighborhood. Specifically, the neighborhood Gaussian aggregation module (NGA) takes a specific pixel as the center and uses Gaussian distribution weights to perform weighted fusion of features in the neighborhood to generate local aggregate features. This local aggregation method can highlight the areas with clear texture and rich details in the feature map, while effectively smoothing the impact of noise on the local area. In this way, the neighborhood Gaussian aggregation module (NGA) enhances the expression ability of local targets at the initial stage of feature extraction, laying a solid foundation for subsequent global optimization.

[0078] On the features processed by the Neighborhood Gaussian Aggregation Module, the global features are further adjusted and optimized by the Feature Self-calibration Module (FCM). The Feature Self-calibration Module learns the relationship between different regions in the feature map through the global self-attention mechanism, generates a global calibration weight matrix, and re-weights the input feature map. On the features processed by the Neighborhood Gaussian Aggregation Module (NGA), the global features are further adjusted and optimized by the Feature Self-calibration Module (FCM). The Feature Self-calibration Module (FCM) learns the relationship between different regions in the feature map through the global self-attention mechanism, generates a global calibration weight matrix, and re-weights the input feature map. Specifically, the Feature Self-calibration Module (FCM) dynamically adjusts the feature distribution of the global background and foreground targets, highlights the important features related to the foreground targets, and suppresses the redundant information of the background area. This global calibration operation can help the model capture the scene context more comprehensively and enhance the ability to distinguish between foreground targets and background areas. In each decoder stage, the Neighborhood Gaussian Aggregation Module (NGA) and the Feature Self-calibration Module (FCM) complement each other and jointly optimize the global and local feature expressions of the input features. The Neighborhood Gaussian Aggregation Module (NGA) enhances the local detail expression of the feature map through Gaussian weighted aggregation, especially the smoothing of boundaries, textures, and noise; while the Feature Self-Calibration Module (FCM) further dynamically adjusts the global background information on this basis to ensure that the model accurately perceives the overall scene. The combination of the two modules enables the decoder to achieve a balance between detail capture and global modeling, which not only effectively reduces noise interference, but also improves the ability to focus on foreground targets in complex scenes. This layer-by-layer optimization strategy provides higher quality feature representation for the generation of multi-level segmentation masks.

[0079] Furthermore, based on the features processed by the neighborhood Gaussian aggregation modeling operation, the input feature map first undergoes global pooling operations in two directions: horizontal pooling and vertical pooling. The global context information is extracted in the horizontal and vertical directions respectively, and two feature vectors are generated. and , and Represent the global feature distribution of the input feature map in the horizontal and vertical directions respectively. Then, the feature vector is broadcasted and Perform element-by-element addition operations to fuse information from two directions and generate a weighted feature map of global concern. After completing global feature fusion, perform strip convolution in two directions on the generated weighted feature map. Use horizontal convolution. Convolution adjusts each row of features to better fit the horizontal shape of the foreground object, and uses Convolution calibrates each column of features to adapt to the vertical shape of the foreground object. After convolution, the weighted features are normalized through batch normalization, and nonlinear transformations are introduced through the ReLU activation function to further enhance the feature response to the foreground area. After completing the calibration of the strip convolution, the feature map undergoes a 3×3 depthwise convolution operation to further extract local detail information. The globally calibrated features are further refined for the edges and small-scale areas in the feature map. Then, the calibrated global attention features are fused with the convolved input feature map through the Hadamard product to weight the calibration information into the original input features. In order to further improve the model's ability to reuse features, the fused features are added to the original input feature map through a residual connection to generate the final output feature map.

[0080] The input feature map first undergoes global pooling operations in two directions: horizontal pooling, ) and Vertical Pooling, ). These two pooling operations extract global context information in the horizontal and vertical directions respectively, generating two feature vectors and , which represent the global feature distribution of the input feature map in the horizontal and vertical directions respectively. Subsequently, the two feature vectors are added element by element through the broadcast mechanism to fuse the information in the two directions and generate a globally focused weighted feature map. This step provides the model with global contextual basic information for capturing and aligning the regions of interest, facilitating more refined calibration and enhancement of subsequent modules.

[0081] After completing the global feature fusion, strip convolution is applied to the generated feature map in two directions: horizontally using Convolution adjusts each row of features to better fit the horizontal shape of the foreground object; vertically, Convolution calibrates each column of features to fit the vertical shape of the foreground object. The role of strip convolution is to decouple the horizontal and vertical calibrations, so that the model can better adapt to foreground objects with complex shapes. After convolution, the features are normalized by Batch Normalization (BN), and nonlinear transformations are introduced through the ReLU activation function to further enhance the response of the features to the foreground area.

[0082] After the strip convolution is calibrated, the feature map undergoes a 3×3 depthwise convolution operation to further extract local detail information. This convolution specifically targets edges and small-scale regions in the feature map, further refining the globally calibrated features. Next, the calibrated global attention features are fused with the convolved input feature map through the Hadamard product (element-wise product), weighting the calibration information into the original input features. Through this fusion mechanism, the model can focus on the region of interest more accurately while suppressing background and noise interference.

[0083] Finally, in order to further improve the model's ability to reuse features, the fused features are added to the original input feature map through residual connections to generate the final output feature map. This step not only retains the low-level feature information of the original input, but also introduces the enhanced effect of the calibrated high-level features. The specific formula is:

[0084] ;

[0085] in, and are the features generated by vertical and horizontal pooling respectively. is the attention feature after strip convolution calibration, is the feature after calibration and fusion, Contains batch normalization BatchNormalization and multi-layer perceptron MLP.

[0086] S4. A cascaded decoder framework is used to perform staged feature extraction on specific blocks output by the image encoder. An independent decoder is used in each stage to align the visual features with the text embedding, and multi-level segmentation masks are gradually generated through feature fusion.

[0087] A cascaded decoder framework is used to perform staged feature extraction on specific blocks output by the image encoder. In the first stage, the image features are first aggregated by the Neighborhood Gaussian Aggregation (NGA) module to aggregate the salient features of the local spatial neighborhood, thereby capturing the spatial relationship of the key area. Subsequently, the image features are self-calibrated by the Feature Self-calibration Module (FCM) to further enhance the semantic consistency of the features. Next, the Text-Image Decoder is used to align the image features with the text embedding through the cross-modal attention mechanism to generate a preliminary segmentation mask. , realizing the fusion and semantic extraction of multimodal information.

[0088] Specifically, the image features and text embedding The image features V processed by the feature self-calibration module and the neighborhood Gaussian aggregation module are input into the text-image decoder, and the input features are standardized by the normalization operation; the normalized text features and image features Input into the relationship description module to generate relationship-enhanced text features , for enhanced text features Through linear transformation, it is mapped to a new representation space; image features Through two independent linear layers, the key K and value V are generated respectively. With key Perform matrix multiplication to generate an attention score matrix, and scale the attention score to reduce the sensitivity of high-dimensional features to attention; perform matrix weighting on the scaled attention score matrix and value V to obtain the output feature after weighted summation , after obtaining the weighted sum of the output features Finally, the model introduces residual connections and layer normalization operations.

[0089] Furthermore, the text-image decoder receives image features and text embedding As input, the image features V processed by the neighborhood Gaussian aggregation module (NGA) and the feature self-calibration module (FCM) are used as input. First, the input features are standardized using the normalization operation, and the calculation formula is:

[0090] ;

[0091]

[0092] ;

[0093] Among them, T is text embedding, g is image feature, V is image feature after being processed by Neighborhood Gaussian Aggregation Module (NGA) and Feature Self-calibration Module (FCM), For normalization operation, is the normalized text feature, is the normalized image feature, is the normalized processed image feature.

[0094] The normalized text features and image features Input into the Relationship Descriptor (RD) module to generate relation-enhanced text features :

[0095] ;

[0096] in, is the text feature, is the image feature, RD is the relationship description module, For enhanced text features.

[0097] Enhanced text features Through linear transformation, it is mapped to the new representation space:

[0098] ;

[0099] in, To enhance the text features, is a linear transformation, is the text feature after linear transformation.

[0100] Image features The key K and value V are generated by two independent linear layers:

[0101] ;

[0102] in, For linear operation, is the image feature, is the key matrix and V is the value matrix.

[0103] After linear transformation With key Perform matrix multiplication to generate the attention score matrix:

[0104] ;

[0105] in, is the matrix multiplication calculation, is the text feature after linear transformation, and K is the key matrix. is the attention score matrix.

[0106] Scale the attention score to reduce the sensitivity of high-dimensional features to attention:

[0107] ;

[0108] in, is the scaling factor, is the attention score matrix. is the scaled attention score matrix.

[0109] right The formula for matrix multiplication of the sum value V is:

[0110] ;

[0111] in, is the matrix multiplication calculation, is the Softmax function, is the scaled attention score matrix and V is the value matrix.

[0112] After obtaining the weighted sum of the output features Finally, the model introduces residual connections and layer normalization operations to ensure that information is not lost when propagating across layers:

[0113] ;

[0114] ;

[0115]

[0116] ;

[0117] in, is the output feature after weighted summation, is the text feature after linear transformation. is the layer normalization operation, is a multi-layer perceptron.

[0118] In the second stage, the input image feature blocks are once again processed by the feature self-calibration module and the neighborhood Gaussian aggregation module to enhance their semantic expression ability and consistency of spatial representation. On this basis, the semantic details are optimized to generate a more refined segmentation mask. , improving the model's ability to depict boundaries and semantically complex areas.

[0119] In the third stage, the enhanced feature self-calibration module and dual-channel decoder are used to further combine the global and local multimodal information on the input image feature blocks to handle the details and semantic consistency issues in complex scenes. The dual-channel decoder deeply fuses the image features and text embeddings to generate the final segmentation mask. The output mask of this stage has highly refined feature representation and multimodal alignment capabilities, representing the optimal segmentation result. ) are the results of independent optimization, but they are not directly used as the input of the next stage. Instead, the cascade decoding framework integrates multi-stage features to uniformly generate the final global segmentation result. Through this integration mechanism, the advantages of segmentation at different stages can be fully utilized, enabling the model to achieve a comprehensive balance between semantic expression and detail description.

[0120] Specifically, the text feature T and the image feature g as well as the image feature V processed by the feature self-calibration module and the neighborhood Gaussian aggregation module are input into the dual-path decoder, and the input features are standardized by the normalization operation; the normalized text feature and image features Input into the relationship description module to generate relationship-enhanced text features , and then the text features Perform multi-head attention operations, then perform residual connection and normalization operations to obtain the normalized and residual connected output ; Normalized and residual connected output And the image features processed by the feature self-calibration module and the neighborhood Gaussian aggregation module As the input of the next multi-head attention, it is then normalized and residually connected to obtain the output of the normalization and residual connection. ; The output of the normalized and residual connections Input the multi-layer perceptron module and then go through normalization and residual connection operations to get the normalized and residual connected output ; The output of the normalized and residual connections Image features processed by the feature self-calibration module and the neighborhood Gaussian aggregation module As the input of the next multi-head attention, it is then normalized and residually connected to obtain the output of the normalization and residual connection. ; The output of the normalized and residual connections Input the multi-layer perceptron module and then go through normalization and residual connection operations to get the normalized and residual connected output ; The output of the normalized and residual connections and Perform matrix multiplication.

[0121] Furthermore, the dual-path decoder first accepts the text features T and image features g as well as the image features V processed by the neighborhood Gaussian aggregation module (NGA) and the feature self-calibration module (FCM) as input, and uses the normalization operation to standardize the input features respectively. The calculation formula is:

[0122] ;

[0123] ;

[0124] ;

[0125] Among them, T is text embedding, g is image feature, V is image feature after being processed by Neighborhood Gaussian Aggregation Module (NGA) and Feature Self-calibration Module (FCM), For normalization operation, is the normalized text feature, is the normalized image feature, is the normalized processed image feature.

[0126] The normalized text features and image features Input into the Relationship Descriptor (RD) module to generate relation-enhanced text features :

[0127] ;

[0128] in, is the text feature, is the image feature, RD is the relationship description module, For enhanced text features.

[0129] The enhanced text features after the relationship description module Perform multi-head attention operations, followed by residual connections and normalization operations:

[0130] ;

[0131] in, For normalization operation, For multi-head attention operation, Enhanced text features after relation description module .

[0132] Normalized and residual connected output And the image features after being processed by the Gaussian Aggregation Module (NGA) and the Feature Self-Calibration Module (FCM) As the input of the next multi-head attention, it is then normalized and connected to the residual:

[0133] ;

[0134] in, For normalization operation, For multi-head attention operation.

[0135] The output is fed into a multilayer perceptron module and then normalized and connected to a residual network:

[0136] ;

[0137] in, For normalization operation, For multi-head attention operation.

[0138] The output is combined with the image features processed by the Gaussian aggregation module (NGA) and the feature self-calibration module (FCM) It is the input of the next multi-head attention, and then goes through normalization and residual connection operation:

[0139] ;

[0140] in, For normalization operation, For multi-head attention operation.

[0141] The output is fed into a multilayer perceptron module and then normalized and connected to a residual network:

[0142] ;

[0143] in, For normalization operation, For multi-head attention operation.

[0144] Will and Perform matrix multiplication operation, the formula is as follows:

[0145] ;

[0146] in, is the matrix multiplication operation.

[0147] S5. Integrate the segmentation masks generated layer by layer to form a final segmentation result, wherein the output segmentation result includes the foreground object category and its pixel-level position distribution.

[0148] Through the layer-by-layer decoder , combining visual features and text embedding to generate segmentation masks at each level. The segmentation masks at each level represent the predictions of different feature levels on the segmentation area, capturing feature information from low-level to high-level;

[0149] The segmentation masks generated layer by layer are integrated through weighted summation, and the results are normalized into the final segmentation prediction using the Softmax function; this process integrates segmentation information at different levels, enabling the model to make full use of multi-scale features and generate more accurate results.

[0150] The difference between the integrated segmentation mask and the true label is calculated through the pixel-level loss function Optimize to form the final segmentation result, where the pixel-level loss function It consists of the Dice Loss function and the Focal Loss function, which enhances the overlap measurement of the foreground area and the robustness to imbalanced samples. The final segmentation result is a probability distribution matrix that contains the target foreground object category and its pixel-level position distribution. This result supports the semantic segmentation task of unseen categories, can efficiently segment the target area in complex scenes, and demonstrates the model's strong zero-sample generalization ability.

[0151] A single-stage zero-shot semantic segmentation method based on CLIP provided in an embodiment of the present invention introduces deep cross visual cue tuning (DCVPT) to alleviate the overfitting problem associated with traditional visual cue adjustment (VPT); the interaction between cue tags and pre-trained tags in the image encoder is promoted through the cross-attention mechanism, and the adaptability to unseen categories is enhanced through residual connections. A single-stage zero-shot semantic segmentation method based on CLIP provided in an embodiment of the present invention also adopts a cascade decoder framework, combines specific blocks of the image encoder for multi-stage feature extraction, and aligns visual features with text embeddings through independent decoders to generate the final segmentation mask, thereby achieving accurate image segmentation. In order to improve the model's ability to pay attention to foreground targets, the present invention also introduces a feature self-calibration module (FCM) to enhance its expressiveness in complex scenes. This method greatly improves the model's ability to recognize unseen categories and has excellent zero-shot generalization capabilities.

[0152] A single-stage zero-shot semantic segmentation method based on CLIP provided in an embodiment of the present invention can effectively solve multiple problems existing in the prior art and improve the performance of the model in zero-shot semantic segmentation tasks. In the prior art, since the image is divided into patches of fixed size, local detail information may be lost, especially when important features are located at the patch boundary. A single-stage zero-shot semantic segmentation method based on CLIP provided in an embodiment of the present invention can effectively enhance the focus on local details of foreground targets by introducing a feature self-calibration module (FCM). The feature self-calibration module (FCM) performs pooling and convolution operations in multiple directions to fine-tune the feature map so that detail information is retained, thereby solving the problem of local detail loss and enabling the model to better perform refined tasks such as small object detection and complex scene segmentation.

[0153] Traditional image encoders usually only rely on the visual features of the last layer, while ignoring the rich object detail information in the middle layer. In order to solve this problem, an embodiment of the present invention provides a single-stage zero-shot semantic segmentation method based on CLIP, and proposes a multi-level feature alignment scheme, which aligns the visual features of the middle layer with the text embedding by designing an independent text-image decoder. In this way, the rich features of the middle layer are fully utilized, the expression ability and detail capture ability of the model are improved, and the cross-modal understanding between images and text is enhanced.

[0154] In the prior art, the fusion of multi-level features may destroy the correlation between vision and language, increase the complexity of the fine-tuning process, and reduce the generalization ability of unseen categories. In order to avoid this problem, the embodiment of the present invention provides a single-stage zero-sample semantic segmentation method based on CLIP, which adopts a cascaded feature fusion method to achieve effective fusion of the middle layer and the last layer of visual features. Through the Gaussian aggregation module (NGA), the feature fusion process is more accurate, effectively avoiding the loss of correlation between features, thereby improving the cross-modal alignment capability of the model and the semantic consistency of visual language.

[0155] In the prior art, the generalization ability of the model for unseen categories is weak, and it usually performs poorly due to the overfitting problem. Although the traditional VPT enhances the discrimination ability, it is easy to cause overfitting, which limits the recognition ability of unseen categories. In order to solve this problem, the embodiment of the present invention provides a single-stage zero-sample semantic segmentation method based on CLIP, a deep cross-visual cue tuning method, which uses a cross-attention mechanism to effectively enhance the interaction between cues and pre-trained labels in the image encoder. Through this design, the model can not only improve the recognition ability of known categories, but also significantly improve the generalization ability of unseen categories, solving the overfitting problem.

[0156] The decoder design in the prior art is complex and single, which easily leads to overfitting during the fine-tuning process. A single-stage zero-sample semantic segmentation method based on CLIP provided in an embodiment of the present invention replaces the traditional decoder architecture by introducing DualPathDecoder, reduces the complexity of the model, and improves the efficiency of the decoding process. Through layer-by-layer decoding and independent optimization, DualPath Decoder avoids the overfitting phenomenon caused by complex decoding structures while maintaining high performance. The stability and generalization ability of the model have been significantly improved, especially when facing unknown categories, it can show stronger robustness.

[0157] The embodiment of the present invention provides a single-stage zero-shot semantic segmentation method based on CLIP, which introduces a deep cross-visual cue tuning method, and optimizes the interaction between cue markers and pre-trained markers in the image encoder through a cross-attention mechanism, thereby effectively alleviating the overfitting problem caused by the traditional visual cue adjustment method and enhancing the generalization ability of the model for unseen categories. At the same time, the present invention also proposes a multi-level feature alignment scheme, which aligns the visual features of the middle layer with the text embedding by designing an independent text-image decoder, fully utilizes the rich feature information of the middle layer, and improves the cross-modal understanding ability between image and text. In addition, the introduction of the cascade feature fusion method and DualPath Decoder ensures the effective fusion of multi-level features, avoids the loss of correlation, reduces the complexity of the model, and significantly alleviates the overfitting phenomenon, thereby improving the stability and generalization ability of the model.

[0158] Based on the same inventive concept, an embodiment of the present application also provides a CLIP-based single-stage zero-sample semantic segmentation device that can execute a processing flow provided by an embodiment of a CLIP-based single-stage zero-sample semantic segmentation method.

[0159] The single-stage zero-sample semantic segmentation device based on CLIP provided in the embodiment of the present application can be used to execute a single-stage zero-sample semantic segmentation method based on CLIP in any of the above embodiments, and its implementation principle and technical effect are similar, which will not be repeated here. The device provided in the embodiment of the present application can be specifically used to execute the scheme provided in the corresponding method embodiment, and the specific functions and technical effects that can be achieved will not be repeated here.

[0160] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0161] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 2As shown, the electronic device may include: a processor 21, a memory 22, and computer program instructions stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program instructions, a CLIP-based single-stage zero-sample semantic segmentation method provided in any of the aforementioned embodiments is implemented.

[0162] Optionally, the above-mentioned components of the electronic device may be connected via a system bus.

[0163] The memory 22 may be a separate storage unit or a storage unit integrated in a processor. The number of processors may be one or more.

[0164] Optionally, the electronic device may further include a communication interface for interacting with other devices.

[0165] It should be understood that the processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0166] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0167] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The above-mentioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the above-mentioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape, floppy disk, optical disc and any combination thereof.

[0168] The electronic device provided in the embodiment of the present application can be used to execute a single-stage zero-sample semantic segmentation method based on CLIP provided in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0169] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed on a computer, the computer executes the above-mentioned CLIP-based single-stage zero-sample semantic segmentation method.

[0170] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0171] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0172] An embodiment of the present application also provides a computer program product, which includes a computer program, which is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, the above-mentioned CLIP-based single-stage zero-sample semantic segmentation method can be implemented.

[0173] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A single-stage zero-shot semantic segmentation method based on CLIP for computer vision processing, characterized in that: The CLIP-based single-stage zero-shot semantic segmentation method includes: S1. The input image is divided into patches of fixed size and fed into the pre-trained CLIP image encoder. Through the deep cross-visual cue adjustment module, the cross-attention of patch labels and cue labels is calculated, and the results are integrated into the visual features to generate enhanced features adapted to unseen categories. S2. Combine the category embedding generated by the text encoder with the global features generated by the image encoder and categorical features The two components are sent to the relationship description module together. The relationship description module generates multi-level classification features through multi-level feature alignment, which are further input into the decoder of the corresponding stage to complete the alignment and enhancement of visual-text features. S3, introduces a feature self-calibration module and a neighborhood Gaussian aggregation module in each decoder to optimize global background extraction and local feature calibration respectively, thereby enhancing the focus on foreground targets while reducing the interference of noise on feature extraction; S4, using a cascaded decoder framework to perform staged feature extraction on specific blocks output by the image encoder, using an independent decoder at each stage to align visual features with text embeddings, and gradually generate multi-level segmentation masks through feature fusion; S5. Integrate the segmentation masks generated layer by layer to form a final segmentation result, wherein the output segmentation result includes the foreground object category and its pixel-level position distribution.

2. The CLIP-based single-stage zero-shot semantic segmentation method according to claim 1, characterized in that: The S1 specifically includes: The input image sample is first divided into fixed-size image patches, each patch contains a fixed pixel block, and the divided image patches are sent to the pre-trained CLIP image encoder to extract patch marker features through preliminary processing of the encoder. ; At each layer of the CLIP image encoder, the patch marker features are first calculated Self-attention to capture correlations between patches within the same image; Mark the text prompt As auxiliary information, and the enhanced patch features Calculate cross attention together to further integrate the information between vision and text; After the cross attention calculation is completed, the fused features are integrated back into the original features through residual connections to ensure that the basic information is retained while enhancing the features. Then, the high-level features are further extracted through the multi-layer perceptron MLP: , where MLP is a multi-layer perceptron. For cross attention, For the visual features after processing; The processed visual features As the input of the subsequent decoder module, it is used to generate multi-level segmentation masks.

3. The CLIP-based single-stage zero-shot semantic segmentation method according to claim 1, characterized in that: The S2 specifically includes: The text encoder generates category embedding based on the input prompt text. The category embedding contains the semantic information of the category. At the same time, the image encoder generates global features and classification features after processing the input image. Among them, the global features reflect the overall semantic information of the entire image, and the classification features contain the detailed details of the local area of ​​the image. The generated category embedding and image features are fed into the relationship description module, which achieves deep fusion of visual and textual information through a multi-level alignment strategy; The enhanced features after multi-level alignment are passed to the cascaded decoder framework in sequence according to a staged design. In each stage, the decoder further utilizes the enhanced features after multi-level alignment to refine the alignment of visual and textual features, and gradually generates multi-level segmentation masks to provide layer-by-layer optimization for the final segmentation results.

4. The CLIP-based single-stage zero-shot semantic segmentation method according to claim 1, characterized in that: The S3 specifically includes: In each decoder, the input features are first subjected to neighborhood Gaussian aggregation. The neighborhood Gaussian aggregation module uses Gaussian distribution weights to perform weighted fusion on the features in the neighborhood with a specific pixel as the center to generate local aggregated features. Based on the features processed by the neighborhood Gaussian aggregation module, the global features are further adjusted and optimized through the feature self-calibration module. The feature self-calibration module learns the relationship between different areas in the feature map through the global self-attention mechanism, generates a global calibration weight matrix, and uses it to re-weight the input feature map.

5. The CLIP-based single-stage zero-shot semantic segmentation method according to claim 4, characterized in that: Based on the features processed by the neighborhood Gaussian aggregation module, the global features are further adjusted and optimized by the feature self-calibration module. The feature self-calibration module learns the relationship between different regions in the feature map through the global self-attention mechanism, generates a global calibration weight matrix, and re-weights the input feature map based on it, specifically including: On the features processed by the neighborhood Gaussian aggregation modeling operation, the input feature map first undergoes global pooling operations in two directions: horizontal pooling and vertical pooling. The global context information is extracted horizontally and vertically respectively to generate two feature vectors. and , and Represent the global feature distribution of the input feature map in the horizontal and vertical directions respectively. Then, the feature vector is broadcasted and Perform element-by-element addition to fuse information from two directions and generate a globally focused weighted feature map. After completing the global feature fusion, the generated weighted feature map is subjected to strip convolution in two directions. Convolution adjusts each row of features to better fit the horizontal shape of the foreground object, and uses Convolution calibrates each column of features to fit the vertical shape of the foreground object. After convolution, the weighted features are normalized by batch normalization, and a nonlinear transformation is introduced through the ReLU activation function to further enhance the feature response to the foreground area. After completing the calibration of the strip convolution, the feature map undergoes a 3×3 depthwise convolution operation to further extract local detail information. The globally calibrated features are further refined for the edges and small-scale areas in the feature map. Then, the calibrated global attention features are fused with the convolved input feature map through the Hadamard product to weight the calibration information into the original input features. To further improve the model's ability to reuse features, the fused features are added to the original input feature map through residual connections to generate the final output feature map.

6. The CLIP-based single-stage zero-shot semantic segmentation method according to claim 1, characterized in that: The S4 specifically includes: A cascaded decoder framework is used to perform staged feature extraction on specific blocks output by the image encoder. In the first stage, the image features are first aggregated by the neighborhood Gaussian aggregation module to aggregate the salient features of the local spatial neighborhood, thereby capturing the spatial relationship of the key area. Subsequently, the image features are self-calibrated by the feature self-calibration module to further enhance the semantic consistency of the features. Then, the image features are aligned with the text embedding through the cross-modal attention mechanism to generate a preliminary segmentation mask. , realize the fusion and semantic extraction of multimodal information; In the second stage, the input image feature blocks are once again processed by the feature self-calibration module and the neighborhood Gaussian aggregation module to enhance their semantic expression ability and consistency of spatial representation. On this basis, the semantic details are optimized to generate a more refined segmentation mask. , improve the model's ability to depict boundaries and semantically complex areas; In the third stage, the input image feature blocks are further combined with global and local multimodal information using the enhanced feature self-calibration module and the dual-path decoder. The dual-path decoder deeply fuses the image features and text embeddings to generate the final segmentation mask. .

7. The CLIP-based single-stage zero-shot semantic segmentation method according to claim 6, characterized in that: The cascaded decoder framework is used to extract features from specific blocks output by the image encoder in stages. In the first stage, the image features are first aggregated by the neighborhood Gaussian aggregation module to aggregate the salient features of the local spatial neighborhood, thereby capturing the spatial relationship of the key area. Subsequently, the image features are self-calibrated by the feature self-calibration module to further enhance the semantic consistency of the features. Then, the image features are aligned with the text embedding through the cross-modal attention mechanism to generate a preliminary segmentation mask. , realizing the fusion and semantic extraction of multimodal information, including: The image features and text embedding The image features V processed by the feature self-calibration module and the neighborhood Gaussian aggregation module are input into the text-image decoder, and the input features are standardized by the normalization operation; The normalized text features and image features Input into the relationship description module to generate relationship-enhanced text features , for enhanced text features Map to the new representation space through linear transformation; Image features Through two independent linear layers, the key K and value V are generated respectively. With key Perform matrix multiplication to generate an attention score matrix and scale the attention scores to reduce the sensitivity of high-dimensional features to attention; Perform matrix weighting on the attention score matrix and value V after scaling to obtain the output feature after weighted summation , after obtaining the weighted sum of the output features Finally, the model introduces residual connections and layer normalization operations.

8. The CLIP-based single-stage zero-shot semantic segmentation method according to claim 7, characterized in that: In the third stage, the input image feature block is further combined with global and local multimodal information using the enhanced feature self-calibration module and the dual-path decoder. The dual-path decoder deeply fuses the image features and text embedding to generate the final segmentation mask. , including: Accept the text feature T and the image feature g as well as the image feature V processed by the feature self-calibration module and the neighborhood Gaussian aggregation module as inputs to the dual-path decoder, and use the normalization operation to standardize the input features respectively; The normalized text features and image features Input into the relationship description module to generate relationship-enhanced text features , and then the text features Perform multi-head attention operations, then perform residual connection and normalization operations to obtain the normalized and residual connected output ; Normalized and residual connected output And the image features processed by the feature self-calibration module and the neighborhood Gaussian aggregation module As the input of the next multi-head attention, it is then normalized and residually connected to obtain the output of the normalization and residual connection. ; The output of the normalized and residual connections Input the multi-layer perceptron module and then go through normalization and residual connection operations to get the normalized and residual connected output ; The output of the normalized and residual connections Image features processed by the feature self-calibration module and the neighborhood Gaussian aggregation module As the input of the next multi-head attention, it is then normalized and residually connected to obtain the output of the normalization and residual connection. ; The output of the normalized and residual connections Input the multi-layer perceptron module and then go through normalization and residual connection operations to get the normalized and residual connected output ; The output of the normalized and residual connections and Perform matrix multiplication.

9. The CLIP-based single-stage zero-shot semantic segmentation method according to claim 1, characterized in that: The S5 specifically includes: Through the layer-by-layer decoder , combining visual features and text embedding to generate segmentation masks at each level. The segmentation masks at each level represent the predictions of different feature levels on the segmentation area, capturing feature information from low-level to high-level; The segmentation masks generated layer by layer are integrated through weighted summation, and the results are normalized into the final segmentation prediction using the Softmax function; The difference between the integrated segmentation mask and the true label is calculated through the pixel-level loss function Optimize to form the final segmentation result, where the pixel-level loss function It is composed of the Dice Loss function and the Focal Loss function, which enhances the overlap measurement of the foreground area and the robustness to unbalanced samples. The final segmentation result is a probability distribution matrix that contains the target foreground object category and its pixel-level position distribution.

10. A single-stage zero-shot semantic segmentation device based on CLIP, characterized in that: The CLIP-based single-stage zero-shot semantic segmentation apparatus is configured to perform the single-stage zero-shot semantic segmentation method according to any one of claims 1 to 9.

Images