Zero-shot image segmentation method based on recursive semantic optimization

Abstract

The application discloses a zero-sample instruction image segmentation method based on recursive semantic optimization, and relates to the technical field of multi-modal image segmentation; the application uses a Segment Anything Model to generate a candidate mask of an image; then the image passes through a constructed visual Transformer backbone network, the visual Transformer backbone network is an image encoder of CLIP, and includes a local branch and a global branch, which respectively extract local features and global context information; then a recursive context attention module is used for aggregating multi-scale context information, enhancing long-distance dependence modeling, a semantic feature refinement module generates a semantic attention map and optimizes visual representation by calculating the similarity of visual features and text features, and finally, the candidate mask and the regional features are combined, and the final segmentation result is obtained through weighted fusion, so that the segmentation accuracy and semantic alignment degree are significantly improved in the zero-sample instruction image segmentation task.

Description

Technical Field

[0001] This invention relates to the field of multimodal image segmentation technology, and more specifically, to a zero-sample indicator image segmentation method based on recursive semantic optimization. Background Technology

[0002] Referencing image segmentation (RIS) is an important task in computer vision, aiming to accurately locate and segment specific regions in an image based on natural language descriptions. Compared to traditional semantic segmentation and instance segmentation, RIS requires fine-grained alignment between visual content and language descriptions, demanding higher cross-modal understanding capabilities. Zero-shot referencing image segmentation can handle unseen language expressions and visual concepts without relying on task-specific annotations, making it an important direction for building scalable and efficient visual language systems.

[0003] The shortcomings of existing technologies include: mismatch between global and local features; visual Transformers often encode global information with category tokens, which makes it difficult to fully represent the local region features corresponding to the text description, thus limiting the segmentation accuracy; and insufficient context modeling capabilities. Standard self-attention is mostly limited to single-layer local context and lacks effective utilization of cross-layer and multi-scale context information, making it difficult to characterize long-distance dependencies in complex scenes, resulting in weak semantic alignment capabilities. When directly using such features for similarity matching, it is difficult to accurately capture the fine-grained correspondence between text descriptions and image regions. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the following solution is proposed to solve the problem of unclear multimodal image segmentation in the above-mentioned background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] The zero-shot indicator image segmentation method based on recursive semantic optimization includes the following steps:

[0007] Step 1: Generate candidate masks for the input image using the Segment Anything Model;

[0008] Step 2: Input the image into the Visual Transformer backbone network, which is the CLIP image encoder to extract multi-scale visual features and attention maps. The backbone network includes local branches and global branches. The local branches extract features of specific regions by multiplying the generated mask with the element-wise product. The global branches process the context information of non-masked regions by Gaussian blurring to preserve global semantics.

[0009] Step 3: Design a recursive context attention module to aggregate multi-scale contextual information to enhance long-range dependency modeling;

[0010] Step 4: Design a semantic feature refinement module to calculate the semantic similarity between visual features and text embeddings, and reconstruct the semantic awareness attention matrix to optimize visual features;

[0011] Step 5: Downsample the generated candidate mask and fuse it with the category tokens of the local branches to extract regional features;

[0012] Step 6: Calculate the region features and the text features after CLIP text encoder, and select the region with the highest similarity mask as the final segmentation result.

[0013] Furthermore, the number of masks generated in step 1 is an adjustable parameter, set between 10 and 100 depending on different application scenarios.

[0014] Furthermore, the dual-branch processing method in step 2 includes:

[0015] The global branch processes the regions outside the mask in the image, and uses Gaussian blur to smooth the non-masked regions;

[0016] The local branch focuses on extracting region features through element-wise multiplication with the generated mask.

[0017] Furthermore, the specific implementation of the recursive context attention module in step 3 includes extracting the intermediate attention map of the original image from the visual Transformer, performing average fusion, and starting from the 9th layer, weighted fusion of the self-attention weight of the current layer with the global context attention.

[0018] Furthermore, the semantic feature refinement module in step 4 includes extracting multi-scale visual features from the visual Transformer, flattening them into spatial feature vectors, projecting them into the text feature space using the CLIP visual projection matrix, calculating the similarity between the projected visual features and the text embedding, generating a semantic graph, and performing weighted fusion.

[0019] Furthermore, the regional feature fusion process in step 5 includes downsampling the generated candidate mask and aligning it with the feature map resolution, then performing mask-weighted pooling on the optimized visual features, generating a feature representation for each region by combining the element-wise product with the mask value, and then performing weighted fusion with the category token features of the local branch.

[0020] Furthermore, the similarity calculation in step 6 uses cosine similarity measurement. The cosine similarity between each region feature and the text features after passing through the CLIP text encoder is calculated, and the mask with the highest similarity value is selected as the final segmentation result.

[0021] The technical effects and advantages of the zero-shot indicator image segmentation method based on recursive semantic optimization in this invention are as follows:

[0022] This invention optimizes the semantic alignment capability of images by calculating the similarity between multi-scale visual features and text embeddings through a semantic feature optimization module and reweighting the visual features. Furthermore, it aggregates multi-level attention maps through a recursive contextual attention module, injecting multi-scale contextual information to enhance global contextual modeling capabilities. This overcomes the limitations of traditional ViT models in long-distance dependency modeling. This optimization based on cross-modal similarity and the effective integration of multi-scale contextual information enables this invention to accurately align image regions with text descriptions in a zero-shot environment, improving segmentation accuracy and accurately identifying complex structures and long-distance dependencies in images.

[0023] Compared with existing zero-shot indicator image segmentation methods based on CLIP and ViT, this method innovatively solves the problems of global-local mismatch, limited context modeling, and weak semantic alignment. It makes up for the limitations of existing methods in fine-grained region segmentation and semantic alignment, and significantly enhances the ability to model long-distance dependencies and multi-scale contextual information in images. By introducing a recursive context attention module and a semantic feature refinement module, it not only improves the accuracy and semantic consistency of image segmentation, but also achieves a breakthrough in segmentation performance in zero-shot environments. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the zero-sample indicator image segmentation method based on recursive semantic optimization of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] In order to achieve the above objectives, Figure 1 A schematic diagram of the zero-sample indicator image segmentation method based on recursive semantic optimization of the present invention is given, which specifically includes the following steps;

[0027] Step 1: Generate candidate masks for the input image using the Segment Anything Model;

[0028] Step 2: Input the image into the Visual Transformer backbone network, which is the CLIP image encoder to extract multi-scale visual features and attention maps. The backbone network includes local branches and global branches. The local branches extract features of specific regions by multiplying the generated mask with the element-wise product. The global branches process the context information of non-masked regions by Gaussian blurring to preserve global semantics.

[0029] Step 3: Design a recursive context attention module to aggregate multi-scale contextual information to enhance long-range dependency modeling;

[0030] Step 4: Design a semantic feature refinement module to calculate the semantic similarity between visual features and text embeddings, and reconstruct the semantic awareness attention matrix to optimize visual features;

[0031] Step 5: Downsample the generated candidate mask and fuse it with the category tokens of the local branches to extract regional features;

[0032] Step 6: Calculate the similarity between the region features and the text description, and select the region with the highest similarity as the final segmentation result. This calculation of the region features and the text description is based on the text features after passing through the CLIP text encoder, and the mask with the highest similarity is selected.

[0033] Step 1, generate candidate masks for the input image using the Segment Anything Model (SAM), including the following steps:

[0034] Candidate masks for the input image are generated using the Segment Anything Model (SAM). The SAM model is a pre-trained deep learning model that automatically generates high-quality image masks without requiring task-specific training. Specifically, the SAM model analyzes the input image, extracts candidate regions related to the target region, and generates corresponding masks. Each candidate mask corresponds to a potential target region in the image, effectively identifying regions of interest within the image.

[0035] The number of generated masks is an adjustable parameter to suit different application scenarios. In practical applications, the number of masks can be adjusted according to the complexity of the image content and the required segmentation accuracy, generally set between 10 and 100. By adjusting the number of masks, the complexity and accuracy of the segmentation task can be flexibly adjusted in different application scenarios.

[0036] In summary, the automation features of the SAM model not only reduce the reliance on task-specific training, but also enable the rapid generation of accurate candidate masks from different image inputs, thereby improving the efficiency and accuracy of image segmentation.

[0037] Step 2: Input the image into the visual Transformer backbone network to extract multi-scale visual features and attention maps. The backbone network includes local branches and global branches. The local branches extract features of specific regions by element-wise multiplication with the generated mask. The global branches process the contextual information of non-masked regions through Gaussian blurring, preserving global semantics. This includes the following steps:

[0038] First, candidate masks for the input image are generated using SAM. Next, the image is further processed through the backbone network of the Visual Transformer (ViT). The Vision Transformer backbone network consists of two parts: a local branch and a global branch, responsible for extracting local features and global contextual information, respectively. This structure is designed to simultaneously capture both detailed and overall semantic information in the image, thereby improving the accuracy of image segmentation and semantic alignment.

[0039] Local branches focus on local regions in the image, defined by candidate masks generated by SAM. Through this design, the model can accurately extract features from the region of interest without being influenced by other parts of the image. Local branches extract features by performing element-wise product with the masks generated by SAM. This process allows for independent feature extraction for each specific region corresponding to each mask, ensuring that the details of that region are preserved. Specifically, for each mask… , with input image Element-wise multiplication yields the local feature representation of the region: ,in, The input is the original image, with a size of , For the first One candidate mask, This indicates an element-wise multiplication operation.

[0040] In this way, local branches can extract high-quality features from specific regions of an image, ensuring the model's sensitivity and accuracy to the target region.

[0041] Global Branch: The global branch is designed to process the contextual information of non-masked regions in an image. These non-masked regions typically contain the image's background or other global information, which is crucial for complete semantic understanding. The global branch processes these regions using Gaussian blur to smooth and preserve the image's global semantic features. Gaussian blur effectively removes detail noise from the image, making the global features smoother and more expressive of contextual semantics. Specifically, the non-masked regions of the image... The region will undergo Gaussian blurring to generate global contextual features: ,in, Indicates the input image Gaussian blur is applied to obtain smooth contextual information of non-masked regions in the image.

[0042] Combining local features with global contextual information: During feature extraction, the local branch and the global branch are responsible for extracting local features and global contextual information of the image, respectively. The fusion of the two in the feature map ensures that the segmentation task not only focuses on the details of the target region, but also retains the semantic information of the background and context. This dual processing of global and local features greatly enhances the model's ability to understand complex scenes, enabling it to maintain high segmentation accuracy and semantic consistency when processing multi-target, complex background and detailed images.

[0043] In summary, the dual-branch structure of the visual Transformer not only effectively extracts multi-scale features from images but also further improves image segmentation performance by combining local and global information. This design ensures that all information in the image can be fully utilized during zero-shot image segmentation, thereby improving segmentation accuracy and semantic alignment capabilities.

[0044] Step 3: Design a recursive context attention module to aggregate multi-scale contextual information to enhance long-range dependency modeling, including the following steps:

[0045] In this step, a Recursive Contextual Attention (RCA) module was designed and implemented. This module is used to aggregate multi-scale contextual information, thereby enhancing the long-distance dependency modeling between global and local features, further improving the image feature transfer capability and the ability to capture fine-grained information. The design goal of the RCA module is to address the limitations of the self-attention mechanism in existing visual Transformers when modeling cross-layer and multi-scale contextual information, especially the inability to effectively capture long-distance dependencies in complex scenes. The specific process includes:

[0046] The steps for extracting the intermediate attention map are as follows:

[0047] The Recursive Context Attention (RCA) module first extracts intermediate attention maps of the original image from layers 5 through 8 of the Visual Transformer network. Through this process, the RCA module can capture attention information of the image at multiple scales, further deepening the understanding of the image content. Specifically, the attention maps... The first from the visual Transformer Layer, in which These intermediate attention maps contain rich contextual information, which helps the model capture semantic features at different scales and levels in the image.

[0048] The extracted intermediate attention maps are aggregated through an averaging fusion operation to form a comprehensive attention map. The diagram contains information from level 5 to level 8: ;

[0049] In this way, the RCA module can fully integrate multi-level contextual information, thereby enhancing the model's ability to model long-distance dependencies.

[0050] The steps for weighted fusion of global and local attention are as follows:

[0051] After extracting the attention maps from the intermediate layers and performing average fusion, the recursive contextual attention module will adjust the self-attention weights of the current layer in layers 9 and beyond of the visual Transformer. Weighted fusion with global contextual attention is performed. The purpose of weighted fusion is to combine the attention information of the current layer with multi-scale contextual information, thereby enhancing the expressive power of features, especially in the transmission of long-distance dependency information in images.

[0052] The weighted fusion operation is implemented using the following formula: ,in, and These are hyperparameters, set to 1.0 and 0.6 respectively. Standard self-attention weights. It is calculated using the query vector and the key vector, and the specific calculation formula is as follows: ,in, It is a query vector. It is a key vector. It is the dimension of the key vector. Through this process, the RCA module effectively integrates the attention weights of the current layer with global context information, further improving the global semantic understanding of the image and the ability to model long-distance dependencies of local features.

[0053] In summary, by designing a recursive context attention module, the model can aggregate global and local information at multiple levels, thereby improving the ability to model long-distance dependencies. Compared with the traditional Transformer model, this module enables the effective transfer of global and local information in the image between different levels through the fusion and weighting of multi-scale information. It demonstrates stronger context modeling capabilities, especially when dealing with complex scenes, multiple objects, or long-distance dependencies.

[0054] Step 4: Design a semantic feature refinement module to calculate the semantic similarity between visual features and text embeddings, and reconstruct the semantic awareness attention matrix to optimize visual features. This includes the following steps:

[0055] In this step, the multi-scale visual features output by the visual Transformer backbone network and the text embeddings are aligned and refined through the semantic feature refinement module to generate a multi-scale semantic map and fuse them to obtain a fused semantic attention map, thereby improving the semantic discrimination ability of image features. The specific process includes the following steps:

[0056] Multi-scale visual feature extraction and flattening:

[0057] Multi-scale visual features are extracted sequentially from layers 5, 6, 7, and 8 of the Visual Transformer. The features of each layer are flattened into spatial feature vectors based on their spatial location. The features of layer l are denoted as: ,in , and These represent the height and width of the feature map for this layer, respectively. As the channel dimension of visual features, this flattening method represents the two-dimensional spatial features of each layer as a set of vectors organized by position, which facilitates unified processing with text features in the future.

[0058] Meanwhile, in order to measure the semantic similarity between visual features and text features within the same feature space, the visual projection matrix in the CLIP model is used to project the above multi-scale visual features onto the text feature space. Let the visual projection matrix be: ,in If is the dimension of the text features, then the projected visual features of the l-th layer are represented as follows: ;in, This represents visual features that have been mapped into the text feature space and are used for similarity calculation with the text embedding.

[0059] Let the text feature embedding be ,in Given the number of tags or semantic units in the text description, the similarity between the projected visual features and the text embedding is calculated to obtain a similarity matrix at each scale. The calculation method is as follows: ,in, This represents the similarity matrix between visual features and text features at the l-th layer. Each element in the matrix describes the semantic similarity between a visual feature and a text feature at a certain spatial location.

[0060] Based on similarity matrix For each scale, a corresponding two-dimensional semantic map is generated. For the l-th layer, at each spatial location, the maximum similarity value between that location and all text embeddings is extracted from the similarity matrix, and this value is reshaped into a two-dimensional semantic map consistent with the spatial resolution of that layer. ,Right now: ;in, , represents the semantic attention graph corresponding to the l-th layer, where the value of each position reflects the highest semantic relevance between that spatial location and the text description.

[0061] In order to integrate semantic information at different levels on a unified spatial scale, semantic graphs of each level are... By upsampling to unify to the maximum spatial size, the corresponding target size is determined by the maximum height and maximum width across all scales, denoted as: For the semantic graph of each layer Upsampling is performed to obtain a semantic map consistent with the maximum scale. Then, weighted fusion is performed according to preset layer weights to obtain the final fused semantic map. Specifically: The weights of each layer are defined as follows: , where l is the layer index, and the weight increases with the layer number, so that deeper features have higher weight in the fusion process, thus making the high-level semantic information more prominent in the final semantic graph.

[0062] Through the above process, step 4 utilizes the similarity relationship between multi-scale visual features and text features to construct a multi-scale semantic map for semantic perception, and obtains a unified fused semantic attention map through weighted fusion. This fused semantic map can highlight image regions that are highly related to the text description in the spatial dimension, providing a more refined semantic basis for subsequent region feature extraction and segmentation result selection.

[0063] Step 5: Downsample the generated candidate mask and fuse it with the category tokens of the local branches to extract region features, including the following steps:

[0064] In this step, the generated candidate mask is downsampled and aligned with the feature map resolution, then weighted and fused with the category token features of the local branches to extract the fused features for each region. The main goal of this step is to combine the region features extracted from the candidate mask with the category token features obtained from the local branches to form the final region feature representation, thereby enhancing the semantic accuracy of each region in the image segmentation task.

[0065] First, the candidate mask generated by SAM is downsampled to ensure that the mask has the same resolution as the feature map. Since the resolution of the candidate mask is usually different from that of the input image, it needs to be adjusted to the same resolution as the image feature map through downsampling. This process ensures that the candidate mask and the image feature map are perfectly aligned in space.

[0066] After downsampling the candidate mask and aligning it with the feature map resolution, feature pooling is performed. The feature pooling process combines element-wise multiplication with the mask value to perform weighted pooling on the features of each region, thereby obtaining the feature representation of each region. The weighted pooling operation ensures that the features within each mask region have appropriate weights during fusion, enhancing the role of region features in the overall segmentation task.

[0067] Specifically, pooling is performed using the following formula: ,in, It is a mask generated by SAM. It is an optimized visual feature representation. This represents element-wise multiplication. Through this operation, the features of a local region are combined with the corresponding mask to extract the feature representation of each region. .

[0068] In addition to region features, the local branch also extracts category token features. This feature represents semantic information of different categories in an image. Category token features play an important role in the semantic classification and region segmentation of images. In this step, the category token feature... Combined with extracted regional features We perform weighted fusion to obtain the final fusion features for each region. .

[0069] Finally, the region features and category token features are weighted and fused to obtain the final fused feature. Fusion coefficient. This controls the contribution ratio between region features and category token features. Specifically, the fusion operation is implemented using the following formula: ,in, This indicates the final fusion characteristics. These are region features extracted from candidate masks. It is a category token feature extracted from a local branch. It is a coefficient that controls the fusion ratio. By adjusting... The value of can control the relative influence of region features and category token features in the final features.

[0070] Through this weighted fusion process, the final fusion feature is obtained It not only includes detailed information about the image region, but also combines category semantic information, providing a more accurate feature representation for subsequent semantic alignment and segmentation result selection.

[0071] Step 6: Calculate the region features and the text features after CLIP text encoder, and select the region with the highest similarity mask as the final segmentation result. This includes the following steps:

[0072] By calculating the similarity between the final fused region features and the text embedding, the image region that best matches the text description is selected as the final image segmentation result. This step uses cosine similarity measurement to evaluate the degree of matching between each region feature and the text description, selecting the most relevant region for final segmentation. The specific process is as follows:

[0073] Similarity calculation: In this step, we first analyze the features of each region. With text embedding Cosine similarity is calculated between the feature vectors to evaluate the semantic match between each region and the text description. The cosine similarity measure measures similarity by calculating the angle between the feature vectors, and its formula is as follows: ,in, Indicates the first The final integration characteristics of each region This indicates the embedding of text descriptions. For the first The similarity between each region and the text description.

[0074] Cosine similarity measures the similarity between region features and text features by calculating the cosine value between them. The similarity value ranges from -1 to 1, with values ​​closer to 1 indicating a better match between the region features and the text description.

[0075] Selecting the most relevant region: Compare all calculated similarity values ​​and select the region with the highest similarity value as the final segmentation result. Specifically, select the region corresponding to the maximum similarity value: ,in, This means selecting the region with the highest similarity value, which will be used as the final segmentation result. In this way, we can ensure that the segmentation result is semantically aligned with the text description and select the image region that best matches the description.

[0076] After selecting the region most relevant to the text description, this region becomes the final image segmentation result. By using cosine similarity measurement and a maximum value selection strategy, this invention can effectively identify the region that best matches the text description from multiple candidate regions, thereby achieving accurate zero-shot image segmentation.

[0077] The method of this embodiment, along with other methods, was compared and evaluated on the RefCOCO, RefCOCO+, and RefCOCOg datasets. Table 1 shows the segmentation accuracy of the method of this invention on the validation and test sets of the RefCOCO, RefCOCO+, and RefCOCOg datasets. The evaluation metrics include the overall intersection-overall reunion ratio (oIoU) and the average intersection-overall reunion ratio (mIoU).

[0078] Experimental results show that the segmentation accuracy of the proposed method on the RefCOCO validation set (val) reaches 42.73% and 50.71% for the overall intersection-union ratio (IU) and mean intersection-union ratio (MURR), respectively; on test set A (testA), it reaches 46.36% and 54.25%, respectively; on test set B (testB), it reaches 38.45% and 45.67%, respectively. On the RefCOCO+ validation set, the segmentation accuracy of the overall IU and mean intersection-union ratio (MURR) reaches 36.00% and 44.98%, respectively; on test set A, it reaches 42.54% and 51.21%, respectively; on test set B, it reaches 30.44% and 37.19%, respectively. On the RefCOCOg validation set, the segmentation accuracy of the overall IU and mean intersection-union ratio (MURR) reaches 42.09% and 51.14%, respectively; and on test set (test), it reaches 42.50% and 51.17%, respectively. This demonstrates that the method of the present invention performs exceptionally well in terms of segmentation accuracy, fully verifying the accurate alignment of image regions and text descriptions in a zero-sample environment.

[0079] Table 1 below compares the experimental results of the method in this embodiment with those of other methods on the RefCOCO, RefCOCO+, and RefCOCOg datasets:

[0080]

[0081] Table 1

[0082] This invention optimizes the semantic alignment capability of images by calculating the similarity between multi-scale visual features and text embeddings through a semantic feature optimization module and reweighting the visual features. Furthermore, it aggregates multi-level attention maps through a recursive contextual attention module, injecting multi-scale contextual information to enhance global contextual modeling capabilities. This overcomes the limitations of traditional ViT models in long-distance dependency modeling. This optimization based on cross-modal similarity and the effective integration of multi-scale contextual information enables this invention to accurately align image regions with text descriptions in a zero-shot environment, improving segmentation accuracy and accurately identifying complex structures and long-distance dependencies in images.

[0083] Compared with existing zero-shot indicator image segmentation methods based on CLIP and ViT, this method innovatively solves the problems of global-local mismatch, limited context modeling, and weak semantic alignment. It makes up for the limitations of existing methods in fine-grained region segmentation and semantic alignment, and significantly enhances the ability to model long-distance dependencies and multi-scale contextual information in images. By introducing a recursive context attention module and a semantic feature refinement module, it not only improves the accuracy and semantic consistency of image segmentation, but also achieves a breakthrough in segmentation performance in zero-shot environments.

[0084] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0085] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0086] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0087] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0088] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A zero-shot indicator image segmentation method based on recursive semantic optimization, characterized in that: The specific steps include: Step 1: Generate candidate masks for the input image using the Segment Anything Model; Step 2: Input the image into the Visual Transformer backbone network, which is the CLIP image encoder to extract multi-scale visual features and attention maps. The backbone network includes local branches and global branches. The local branches extract region features by multiplying the generated mask element-wise, while the global branches process the context information of non-masked regions through Gaussian blur to preserve global semantics. Step 3: Design a recursive context attention module to aggregate multi-scale contextual information to enhance long-range dependency modeling; Step 4: Design a semantic feature refinement module to calculate the semantic similarity between visual features and text embeddings, and reconstruct the semantic awareness attention matrix to optimize visual features; Step 5: Downsample the generated candidate mask and fuse it with the category tokens of the local branches to extract regional features; Step 6: Calculate the region features and the text features after CLIP text encoder, and select the region with the highest similarity as the final segmentation result. The specific implementation of the recursive context attention module in step 3 includes extracting the intermediate attention map of the original image from the visual Transformer, performing average fusion, and starting from the 9th layer, weighted fusion of the self-attention weight of the current layer with the global context attention. The semantic feature refinement module in step 4 includes extracting multi-scale visual features from the visual Transformer, flattening them into spatial feature vectors, projecting them onto the text feature space through the CLIP visual projection matrix, calculating the similarity between the projected visual features and the text embedding, generating a semantic graph and performing weighted fusion. The region feature fusion process in step 5 includes downsampling the generated candidate mask and aligning it with the feature map resolution, then performing mask-weighted pooling on the optimized visual features, generating a feature representation for each region by combining the element-wise product with the mask value, and then performing weighted fusion with the category token features of the local branch.

2. The zero-shot indicator image segmentation method based on recursive semantic optimization according to claim 1, characterized in that: The number of masks generated in step 1 is an adjustable parameter, which can be set between 10 and 100 depending on the application scenario.

3. The zero-shot indicator image segmentation method based on recursive semantic optimization according to claim 2, characterized in that: The two-branch processing method in step 2 includes: The global branch processes the regions outside the mask in the image, and uses Gaussian blur to smooth the non-masked regions; The local branch focuses on extracting region features through element-wise multiplication with the generated mask.

4. The zero-shot indicator image segmentation method based on recursive semantic optimization according to claim 3, characterized in that: The similarity calculation in step 6 uses cosine similarity measurement. The cosine similarity between each region feature and the text features after passing through the CLIP text encoder is calculated, and the mask with the highest similarity value is selected as the final segmentation result.

Images