Sam2 small sample segmentation method based on semantic-visual dual memory fusion

Abstract

The application discloses a SAM2 small sample segmentation method based on semantic-visual double memory fusion, constructs semantic query memory, visual query memory and query related support visual memory, fuses the semantic query memory and the visual query memory through a memory refinement module guided by the query related support visual memory, and combines SAM2 dense matching and decoding module end-to-end training. The method solves the single memory and foreground-background confusion problems of the existing method, enhances target semantic consistency and fine-grained modeling capability, significantly improves segmentation precision and generalization capability in a complex scene, and is suitable for medical image analysis, automatic driving and the like.

Description

Technical Field

[0001] This invention relates to the field of computer vision and machine learning, specifically to a SAM2 few-shot segmentation method based on semantic-visual dual-memory fusion. Background Technology

[0002] In recent years, semantic segmentation, as one of the core tasks in computer vision, has received widespread attention in both academic research and industrial applications. Deep learning-based semantic segmentation methods (such as fully convolutional networks and U-Net architectures) have made significant progress and demonstrated excellent performance in various scenarios. However, the success of these methods highly depends on the availability of large-scale, high-quality labeled data. For semantic segmentation tasks, the cost of manual annotation is extremely high due to the need for precise annotation of every pixel in an image, resulting in a severe shortage of labeled data in real-world scenarios. In this situation, directly training deep models can easily lead to overfitting, causing a significant decrease in the model's generalization ability when segmenting unknown categories or complex scenes. To address the problem of scarce labeled data, few-shot semantic segmentation methods have been proposed. Furthermore, with the development of fundamental visual models, these models provide fine-grained visual representations, making few-shot semantic segmentation easier.

[0003] Current leading-edge methods are based on the SAM2 model. However, SAM2 was originally designed for video tasks, and its core mechanism relies on a key assumption: memory frames and the current frame come from the same continuous video segment, thus possessing natural continuity and alignment in terms of temporal sequence and appearance. When SAM2 is transferred to few-shot segmentation scenarios, the "memory" required by the model comes from independent and limited support images, which often differ significantly from the query images in terms of viewpoint, lighting, background, and object pose. This fundamental inter-domain difference leads to a severe support-query mismatch problem, making it difficult for SAM2 to directly utilize irrelevant visual memory for effective inference.

[0004] To mitigate this problem, existing research primarily attempts to bridge the gap between support and query by generating intermediate transition images, predicting query-guided hints, or adaptively adjusting the feature space. While these methods have made some progress on specific datasets, they often fail to generate or convert high-quality memories that are highly aligned with and semantically consistent with the query image when faced with challenging samples exhibiting extreme differences in visual appearance. Consequently, SAM2's inference based on this memory may fail, ultimately leading to decreased segmentation accuracy, particularly in object boundaries, details, and novel categories. Summary of the Invention

[0005] Addressing the shortcomings of existing technologies, this invention primarily aims to solve the problem of achieving high-precision image semantic segmentation with a small number of samples in the intersection of computer vision and machine learning, particularly in scenarios where labeled data is scarce. Furthermore, to further address intra-class diversity and bridge the gap between support and query data, this invention provides a SAM2 small-sample segmentation method based on semantic-visual dual-memory fusion. This method constructs semantic query memory, visual query memory, and query-related support visual memory. A memory refinement module guided by the query-related support visual memory fuses the semantic and visual query memories. Combined with end-to-end training using the SAM2 dense matching and decoding module, this improves the model's generalization ability to unknown categories. It can be widely applied in fields requiring rapid adaptation to new category segmentation tasks, such as medical image analysis and autonomous driving environmental perception.

[0006] To explore the essence of optimal memory, this invention first conducted a basic control experiment: referring to FSSAM, memories for SAM2 were constructed using only textual and visual cues. The experimental results showed that text-driven memory significantly outperformed purely visual memory. This finding led to a deeper analysis: textual information tends to activate regions in an image related to semantic concepts, while visual information focuses more on activating regions with similar appearances. Intuitively, the regions activated by these two methods should differ significantly; semantic memory is more conservative, while visual memory is more aggressive.

[0007] Semantic memory, being overly conservative, leads to undersegmentation; visual memory, being overly aggressive, results in oversegmentation. Based on this, this invention naturally proposes a solution: can we use conservative semantic memory as the main component, while simultaneously supplementing it with aggressive visual memory, and use this supplementary process of supporting set information calibration? Through this synergy and calibration, this invention aims to generate high-quality memories that combine accuracy (from semantics) and completeness (from vision), thereby driving SAM2 to achieve more precise segmentation.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] This invention provides a few-sample semantic segmentation method that utilizes the complementary mechanism of semantic memory and visual memory to optimize the SAM2 memory encoding and reasoning process, comprising the following steps:

[0010] S1: Construct the input for the few-sample segmentation task.

[0011] The query image to be segmented, several support images, and their corresponding support sample masks are obtained. The above data are uniformly resized, normalized, and standardized to form preprocessed support-query image pairs. The text information of the pairs is constructed according to the segmentation category for subsequent feature extraction and memory construction.

[0012] S2: Construct semantic query memory.

[0013] The preprocessed query image and text information are input into a pre-trained visual-language model, which, combined with semantic descriptions of the target category, generates a semantic response map of the target region. This memory focuses on expressing the stable relevance of the target in the semantic space, providing reliable foreground prior information.

[0014] S3: Build visual query memory.

[0015] The support and query images are input together into the DINOV2 image encoder to extract high-dimensional visual features. Using a support mask, a pixel-level visual similarity map is calculated based on the cross-image correlation between support and query samples to extract potential foreground regions and construct a visually enhanced query memory representation. This memory can capture more potential target regions, but may also introduce some noise to improve target coverage.

[0016] S4: Build visual memory to support query-related queries.

[0017] Using the cross-image correlation between the query semantic response map obtained in step S2 and the support samples, a pixel-level visual similarity map is calculated to extract potential foreground regions. Combined with foreground feature regions defined by labeled masks in the support images, a query-related support visual memory representation is constructed. The resulting support visual memory is not a direct copy of the support samples, but rather an adaptive reconstruction result under query constraints, enabling it to more accurately reflect the category attributes and geometric shape of the query target.

[0018] S5: Based on memory-guided query memory fusion.

[0019] Semantic memory, query visual memory, and supporting visual memory are input into a guided memory refinement module for joint processing. The supporting visual memory serves as guiding information to calibrate and constrain the query visual memory. By enhancing the response of the target foreground region and suppressing interference information in the background region, the query memory is made more focused on effective semantic and visual features related to the target category, thereby obtaining a more stable and discriminative refined query memory representation, which is then used to generate subsequent target segmentation results.

[0020] S6: End-to-end joint training.

[0021] The effective memory obtained in step S5, along with the visual features of the query image, are input into the memory attention module and mask decoding module of SAM2, ultimately outputting the predicted segmentation result of the query image. A binary cross-entropy loss function is used for pixel-level supervision between the obtained predicted segmentation mask and the ground truth labeled mask of the query image. Based on this loss, backpropagation and parameter updates are performed on the entire network model, achieving end-to-end joint optimization of all sub-modules.

[0022] Preferably, to accurately locate salient regions in the query image that are highly correlated with the target class at the semantic level, thereby providing reliable guidance for subsequent memory construction and feature enhancement, this invention generates semantic prompts on the query image based on the PICLIP method to construct robust query semantic memory. Specifically, a set of text prompts describing target and non-target concepts is first constructed, in the forms of "a photo of {target class}" and "a photo without {target class}". These text prompts are input into the CLIP text encoder to obtain the corresponding foreground and background text features, denoted as follows: Given a query image Visual features are extracted using the CLIP visual encoder. (Remove class tokens) and generate a global query vector for the query image using global average pooling: Next, cosine similarity is used to calculate the matching degree between the query vector and text features, and then normalized using the softmax function to obtain the semantic response scores of the foreground and background.

[0023]

[0024] in, The temperature coefficient is used. Based on this, the present invention employs a softmax-GradCAM mechanism to backpropagate the gradient from the final classification score to the first LayerNorm layer in the last Transformer block of the CLIP visual encoder, and averages it spatially to obtain the semantic relevance weights of each channel:

[0025]

[0026] in, Indicates the selected layer number Feature map of each channel This represents the spatial location index. Next, the feature maps of each channel are weighted and summed according to their corresponding semantic weights, and negative responses are filtered out using the ReLU activation function to obtain the semantic cue map of the query image.

[0027]

[0028] The semantic cue map highlights visual regions highly relevant to the target text cue, thus forming a region response that is semantically highly aligned with the target category. Finally, the semantic cue map and the SAM features of the query image are processed by the memory encoder in SAM2 to obtain the semantic memory representation of the query, denoted as: .

[0029] Preferably, to enhance the perception of the appearance and details of the target in the query image, the present invention establishes visual cues based on visual representation in step S3 to supplement the deficiencies of semantic cues in terms of fine-grainedness and boundary information. The specific implementation steps are as follows: First, the DINOv2 visual basic model is used to analyze the query image. With supporting images Feature extraction is performed to obtain the corresponding high-dimensional visual feature mapping:

[0030]

[0031] in (or rearrange as needed) ), For channel dimension, For spatial resolution. Secondly, based on the foreground annotation mask of the supporting image. Masked average pooling is performed on the supporting features to obtain the foreground prototype of the supporting image:

[0032]

[0033] The prototype is a length of The vectors, which centrally represent the overall appearance and semantic information of targets in the supporting samples, can be used for appearance similarity comparison with the query location. Subsequently, foreground prototypes will be supported. DINO features of the query image Similarity is calculated at the pixel level, and normalized cosine similarity is used to measure the degree of matching between each pixel position and the supporting foreground, resulting in a query visual cue image. :

[0034] .

[0035] here This represents the cosine similarity calculated per pixel (by taking the cosine of the feature vector at each spatial location with the prototype), and the result is a similarity map that corresponds one-to-one with the query pixel. This indicates that min–max normalization is performed on the similarity map, which compresses the range of the hint map values ​​to a smaller value. This facilitates subsequent fusion and thresholding processing. Explicitly written in component form:

[0036] .

[0037] The query visual prompts obtained in this invention Spatially, it directly indicates the region in the query image that most closely matches the supporting foreground in appearance. Finally, the SAM features of this visual cue image and the query image are processed by the memory encoder in SAM2 to obtain the semantic memory representation of the query, denoted as: .

[0038] Preferably, after obtaining semantic query memory Visual query memory And visual memory support related to queries Subsequently, to further enhance the consistency of the query foreground and suppress interference from the background region, this invention introduces a memory fusion mechanism based on cross-attention and foreground prototype guidance in step S5 to adaptively calibrate and enhance the query memory. First, the three memory mappings mentioned above are projected onto a unified latent space through a linear layer to construct the query vector. Key vector Value vector and support key vectors :

[0039]

[0040] in, This represents the number of spatial locations in the feature map. To hide the size of the dimension, This represents a learnable linear mapping. Subsequently, self-attention weights are first established within the query, calculated... and The scaled dot product between them yields the query self-attention matrix:

[0041]

[0042] in Represents matrix multiplication. Let be the dimension of the key vector. The self-attention matrix reflects the autocorrelation of query features in the spatial dimension. Meanwhile, to further strengthen the correspondence between foregrounds (FG–FG), the most relevant foreground prototypes to the query are extracted from the support memory. Specifically, the support memory... Supported by visual cues A globally weighted average is applied over the guided foreground region to obtain the query-related supporting foreground prototype:

[0043]

[0044] Next, the supporting foreground prototype is calculated. With query vector Cosine similarity at each spatial location is used to construct foreground enhancement guiding weights:

[0045]

[0046] in To prevent small constants from causing numerical instability, the obtained similarity matrix... This reflects the degree of matching between the supporting foreground and the query positions. To further highlight the foreground region and weaken the response intensity of the background region, this invention normalizes and merges the two attention maps mentioned above, introducing a background suppression mechanism:

[0047]

[0048] And further updated to:

[0049]

[0050] in Represents the normalization operation, symbol This means that only the negative values ​​are retained to suppress the background area. A scaling factor is used to control the suppression intensity (empirically set to 10 in this embodiment). After foreground enhancement and background suppression are completed, the updated attention weight matrix is... Perform softmax normalization and use it on the query value vector. By performing weighted aggregation, we obtain the enhanced query representation:

[0051]

[0052] Among them, residual joins are used to memorize the original semantic query. This is added to the enhanced results, thereby injecting further foreground enhancement information guided by supporting samples while preserving the original semantic structure.

[0053] The enhanced query memory obtained through the above steps It is more spatially focused on the target region, more semantically consistent with the supporting foreground, and effectively suppressed in the background region, providing a more reliable and discriminative memory representation for subsequent input to the segmentation model (SAM2 decoder).

[0054] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described few-sample image segmentation method based on a prototype-affinity hybrid network.

[0055] The technical principle of this invention is as follows:

[0056] This invention proposes a few-shot semantic segmentation technique that simulates the human ability to quickly learn new concepts from a small number of examples. It utilizes a small number of labeled samples (called support samples) to extract prior knowledge of the target object, thereby achieving efficient segmentation of similar objects in unlabeled images (called query samples). In recent years, with the development of basic visual models, the Segment Anything Model (SAM) and its upgraded version SAM2 have further enhanced the model's generalization ability in multi-object and multi-scene scenarios by introducing a more sophisticated memory encoding and matching mechanism. SAM2 can not only handle single-image segmentation tasks but also model and utilize historical information through a memory encoder and dense feature matching module, providing important prior support for target segmentation in complex scenes. Therefore, SAM2 provides a highly promising technical path for introducing memory mechanisms into few-shot semantic segmentation tasks.

[0057] Based on the analysis of the current state of few-sample semantic segmentation technology, this invention aims to solve the following key problems caused by the inherent defects of the SAM2 model in existing methods:

[0058] 1. SAM2's video memory mechanism is mismatched with the memory of few-shot segmentation tasks. SAM2 relies on temporally continuous and visually consistent video frames to build memory, while in few-shot segmentation, the support images and query images are independent and significantly different, leading to a severe "memory mismatch".

[0059] 2. The memory construction method is singular, making it difficult to take into account both semantic and detailed information. When constructing memories, the existing SAM2 model usually relies on a single form of mask or visual feature as the memory source. Although this method can provide some prior information about the target, this single-source memory representation can only highlight the general area of ​​the target, but it is difficult to accurately depict the true boundary and fine structure of the target, thus limiting the segmentation accuracy of the model in scenarios with few samples.

[0060] 3. Insufficient ability to distinguish foreground from background, easily introducing redundant information. Due to the extremely limited available labeled information under conditions of few samples, the memory constructed by SAM2 often contains too much background region information, especially in complex scenes or when the background and target have high similarity, making it difficult for the model to accurately distinguish between foreground and background. This can lead to some background regions being mistakenly identified as foreground, resulting in false detections or oversegmentation, affecting the final segmentation results.

[0061] The purpose of this invention is to address the shortcomings of existing SAM2 methods in few-shot semantic segmentation scenarios, such as coarse memory construction, foreground-background confusion, and blurred boundaries. This invention proposes a SAM2 few-shot segmentation method based on semantic-visual dual-memory fusion. This method constructs multi-source memories with complementary semantic and visual features, enhancing both the semantic consistency of the target and the fine-grained structural modeling capability, thereby achieving a synergistic improvement in segmentation accuracy and robustness in complex scenarios.

[0062] The advantages and beneficial effects of this invention are as follows:

[0063] 1. This invention is the first to systematically study which type of memory information is most effective in improving the performance of SAM2 in few-sample segmentation tasks. Through comparative analysis, it was found that SAM2 has a significant preference for "conservative" memory cues. This conclusion provides an important theoretical basis for subsequent memory construction strategies, thereby reducing the occurrence of error augmentation and mismatch problems at the source.

[0064] 2. This invention designs a semantic-visual cue generation module, which can extract semantic and visual cues with complementary characteristics from the supporting image and the query image respectively, thereby constructing a multi-type memory representation that simultaneously contains conservative and radical information, enabling the model to have stronger perception and adaptability while maintaining stability.

[0065] 3. The present invention further proposes a support calibration memory refinement module, which uses the discriminative information in the support samples to supplement and calibrate the conservative memory in a targeted manner, thereby making up for the lack of information in the details and boundary regions of the single conservative memory, so that the final memory representation is more complete and accurate, and provides more reliable constraints for subsequent segmentation.

[0066] 4. Through the synergistic effect of the above modules, this invention can achieve segmentation performance superior to existing methods on public datasets such as COCO and PASCAL. It can also maintain high stability and accuracy in real-world scenarios with complex backgrounds and varied target appearances, significantly improving the generalization ability and practical application value of the few-shot segmentation model. Attached Figure Description

[0067] Figure 1 This is a network structure diagram of the method of the present invention;

[0068] Figure 2 This is a schematic diagram illustrating the detection effects of existing methods and the method of this invention;

[0069] Figure 3 The results show the performance comparison between existing methods and the method of this invention on the PASCAL dataset;

[0070] Figure 4The results show a performance comparison between existing methods and the method of this invention on the COCO dataset.

[0071] Figure 1 In Chinese: (a) Semantic memory building module (semantic query memory building module), (b) Visual memory building module (visual query memory, query-related support visual memory building module), (c) memory refinement module with support for guidance; Figures 2-4 In Chinese: ours refers to the method of this invention, and FSSAM refers to the existing method. Detailed Implementation

[0072] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] like Figure 1 As shown, a SAM2 few-shot segmentation method based on semantic-visual dual-memory fusion includes the following steps:

[0074] S1: Data Preprocessing and Support-Query Pair Construction. Input data includes the query image to be segmented, support images, and their corresponding binary masks. Support images provide annotation information for the target category, and the masks accurately mark the foreground target; the query image is the object to be segmented, and its mask needs to be predicted by the model. In the data preprocessing stage, images are standardized and enhanced: images are uniformly scaled to 512×512 resolution, pixel values ​​are normalized to the range [0,1], and random horizontal flipping, random cropping, and color dithering are performed. Task pairs are constructed by randomly sampling support images and query images to simulate a few-sample segmentation scenario, ensuring no overlap between categories during training and testing, thereby verifying the model's generalization ability to unknown categories.

[0075] S2: Construction of Query Semantic Memory Based on Semantic Hints. This step aims to acquire high-level information that clearly indicates the target category from a semantic level, thereby constructing a semantic query memory. First, based on the current task category, at least two types of text hints are constructed to describe semantic concepts containing and not containing the target category, respectively. For example, text hints can be expressed as semantic descriptions such as "image containing the target object" and "image not containing the target object." Next, the above text hints are input into a pre-trained text encoding model (such as CLIP text encoder) to obtain the corresponding semantic feature representations. Simultaneously, the query image is input into a visual encoding model to obtain its corresponding visual feature embedding representation, and global features are extracted from it. Subsequently, the similarity between the query image features and different text semantic features is calculated to determine the strength of the relevance between the query image and the target concept. The contribution of each spatial location to the final semantic judgment result is obtained through a backpropagation mechanism, thereby generating a semantic relevance map. This semantic relevance map can highlight the regions in the query image that are highly semantically related to the target category and suppress parts that are irrelevant to the target. Finally, the semantic relevance map is fused with the deep features of the query image to form the query semantic memory. This memory provides strong semantically consistent guiding information on a global scale.

[0076] S3: Construction of Query Visual Cues Based on Support Image Foreground. This step, from the perspective of visual similarity, utilizes foreground information in the support samples to locate potential similar target regions in the query image to construct query visual cues. Specifically, firstly, the support image and its corresponding foreground mask are input into a visual feature extraction model (such as DINOv2) to extract high-discrimination deep features. Using the foreground mask of the support image, the foreground regions in the support feature map are aggregated to extract a support foreground prototype that can represent the overall appearance features of the current target category. Subsequently, the query image is also input into the same visual feature extraction model to obtain the feature representation of the query image at various spatial locations. Next, these query features are compared with the support foreground prototype position by position to form a visual similarity map. This visual similarity map can highlight the regions in the query image that are highly similar to the support foreground in appearance features, thus providing a salient visual region cue for the query image in the spatial dimension, which is called the query visual cue. Finally, the query visual cue is fused with the deep features of the query image to form a visual query memory.

[0077] S4: Constructing Query-Relevant Supporting Visual Memory. After obtaining query semantic cues and query visual cues, this step further explores the deeper correspondence between the query image and the supporting images to construct query-relevant supporting visual memory. First, the regions highlighted by the query semantic cues in the query image are extracted as potential target regions. Simultaneously, their global visual feature representations are extracted from the supporting images. Next, through cross-image correlation analysis, the similarity relationship between the potential foreground region of the query and various locations in the supporting images is calculated, obtaining a pixel-level correlation map across images. Using this correlation map, parts of the supporting images that are highly visually and semantically related to the query foreground region are selected to form a set of dynamically selected support features. By further integrating and encoding these support features, query-relevant supporting visual memory is constructed. Unlike traditional fixed support features extracted only using support masks, this memory can be dynamically adjusted according to the specific query image, more closely matching the true appearance features of the target to be segmented.

[0078] S5: Cross-modal memory fusion based on foreground consistency enhancement and background suppression. After obtaining query semantic memory, query visual memory, and query-related supporting visual memory, this step introduces an attention fusion module with foreground consistency constraints and background suppression mechanisms to jointly model multi-source memories, further improving the salience of the target region and reducing background interference. First, the memories from different sources are mapped to a unified feature space, establishing correspondences between memories in the spatial dimension. Next, a spatial correlation strength map is constructed within the query memory to capture the degree of association between regions in the query image, enhancing the highly consistent region responses in query semantic memory and query visual memory. Simultaneously, for regions with low similarity to the supporting foreground, their weight in the attention map is reduced through suppression mechanisms, thereby achieving background suppression.

[0079] After foreground enhancement and background suppression, the weighted result is fused with the original query semantic memory. Residual connections are used to retain the original semantic information while introducing additional discriminative enhancement information, ultimately yielding an enhanced query memory representation. This enhanced query memory not only has clear semantic directionality but also focuses more spatially on the target region and more closely approximates the foreground distribution of supporting samples at the feature level, providing a more reliable basis for subsequent segmentation.

[0080] S6: Target segmentation prediction based on enhanced query memory. Finally, the enhanced query memory obtained in step S5 is used as cue information and input into the segmentation decoding model (SAM2 segmentation network) to guide the final target segmentation. During the segmentation process, the model uses the enhanced query memory as contextual guidance information to determine whether a pixel in the query image belongs to the target region, thereby generating a complete segmentation prediction map. Compared to directly using the original features for segmentation, this invention uses enhanced memory obtained through multi-source information fusion, enabling the model to have stronger response capabilities in the target region and stronger suppression capabilities in non-target regions, thus effectively improving the accuracy, boundary integrity, and robustness to complex backgrounds of the segmentation results. The final output is the accurate segmentation mask of the target in the query image.

[0081] Example 1

[0082] Data preparation: Select the PASCAL-5i dataset, set the task according to 5-way 1-shot / 5-shot, and ensure that the training and test images have no overlap in categories; during preprocessing, the images are scaled to 512×512, the pixels are normalized to [0,1], randomly flipped horizontally (probability 0.5), randomly cropped (size 384×384), and color jittered (brightness ±0.1, contrast ±0.1).

[0083] Semantic query memory construction: Construct text prompts "a photo of a person" and "a photo without a person", input them into the CLIP text encoder to obtain foreground features. Background features Features are extracted from the query image using the CLIP visual encoder. (d=512), generate global query vector Set the temperature coefficient τ = 0.07 and calculate the semantic response score. , Semantic cue graphs are generated using softmax-Grad CAM. The semantic memory is obtained by combining the SAM features with the memory encoder. .

[0084] Visual query memory construction: DINOv2 extracts features from supporting and query images. (d=768), the supporting foreground prototype is obtained through masked average pooling. ; Calculate the query pixel and The cosine similarity is used to generate visual cue maps after min-max normalization. Visual memory is obtained by combining SAM features. .

[0085] Supports visual memory construction: based on Extract potential foreground regions from the query, calculate cross-image correlations with supporting images, filter relevant supporting features, and construct support for visual memory. .

[0086] Memory fusion: Set the hidden dimension C=256, project multi-source memory onto the linear layer to obtain Q, K, V, ;calculate Support for foreground prototypes Obtained by global weighted average; (ε=1e-8); Fusion Retain the negative values, α=10; weighted V after softmax normalization, residual connection. Enhanced memory .

[0087] Training and Segmentation: Input the SAM2 decoding module, use the binary cross-entropy loss function (weights 1.0), and the AdamW optimizer (learning rate 1e-3, weight decay 1e-5), train for 50 epochs; output the segmentation mask during testing, and analyze it qualitatively, such as... Figure 2 As shown in the left half, the method of this invention can completely and accurately segment the target category; from a quantitative perspective, such as Figure 3 As shown, PASCAL-5i achieves an average accuracy of 85.0% for 1-shot and 87.0% for 5-shot, representing improvements of 4.0% and 1.6% respectively compared to the previous best-performing method.

[0088] Example 2

[0089] Data preparation: Select the COCO-20 dataset, set it to 5-way 1-shot / 5-shot, add random rotation (0-10°) to the preprocessing, scale to 512×512, and normalize to [0,1].

[0090] Semantic query memory optimization: Expand the text prompts to richer semantics ("a photo of a person with clothes, people, human" "a photo without a person with clothes, people, human") to improve semantic robustness; the CLIP model adopts the ViT-L / 14 version with a feature dimension d=768.

[0091] Visual feature extraction optimization: DINOv2 uses the vit-large model, and the feature map resolution is adjusted to 14×14 to enhance the ability to capture details.

[0092] Memory fusion optimization: The scaling factor α is adaptively adjusted (dynamically set to 5-15 based on the foreground-background ratio within the batch) to improve adaptability to complex backgrounds; layer normalization is introduced to process Q, K, and V to stabilize the training process.

[0093] Training and Segmentation: The loss function used is binary cross-entropy + Dice loss (weight ratio 1:0.5), the learning rate is 1e-4, and the training lasts for 10 epochs; from a qualitative perspective, such as... Figure 2 As shown in the right half, the method of this invention can completely and accurately segment the target category; from a quantitative perspective, such as Figure 4 As shown, the COCO-20i 1-shot average accuracy is 67.3% and 5-shot accuracy is 70.2%, which is significantly better than the existing FS-SAM2 method.

Claims

1. A SAM2 few-shot segmentation method based on semantic-visual dual-memory fusion, characterized in that: Includes the following steps: S1. Input for constructing a few-sample segmentation task: Obtain the query image to be segmented, several support images and their corresponding support sample masks. Perform uniform size adjustment, normalization and standardization on the above images to form preprocessed support-query image pairs. Construct paired text information according to the segmentation category for subsequent feature extraction and memory construction. S2. Constructing Semantic Query Memory: Input the preprocessed query image and text information into the pre-trained visual-language model, combine it with the semantic description information of the target category, and generate a semantic response map of the target region; this memory focuses on expressing the stable relevance of the target in the semantic space, and is used to provide reliable foreground prior information; S3. Constructing Visual Query Memory: Input the supporting image and the query image into the DINOV2 image encoder to extract high-dimensional visual features; By leveraging support masks based on cross-image correlation between support samples and query samples, pixel-level visual similarity maps are calculated to extract potential foreground regions and construct visually enhanced query memory representations. This memory can capture more potential target areas, thereby improving target coverage; S4. Construct query-related supporting visual memory: Utilize the cross-image correlation between the query semantic response map obtained in step S2 and the supporting samples to calculate the pixel-level visual similarity map, extract the potential foreground region on the supporting image, and combine it with the foreground feature region in the supporting image defined by the label mask to construct the query-related supporting visual memory representation. The obtained supporting visual memory is not a direct copy of the supporting samples, but an adaptive reconstruction result under the constraints of the query conditions, which enables it to more accurately reflect the category attributes and geometric shape of the query target. S5. Query memory fusion guided by support memory: Semantic query memory, visual query memory, and support visual memory are input into the support-guided memory refinement module for joint processing. In order to enhance the consistency of the query foreground and suppress the interference of the background region, the foreground prototype most relevant to the query is extracted from the support visual memory. A memory fusion mechanism based on cross attention and foreground prototype guidance is introduced to adaptively calibrate and enhance the query memory. The enhanced result is combined with the original semantic query memory through residual connection to obtain the enhanced query memory representation. S6. End-to-end joint training: The enhanced query memory obtained in step S5 and the visual features of the query image are jointly input into the memory attention module and mask decoding module of SAM2, and finally output the predicted segmentation result of the query image. The obtained predicted segmentation mask and the ground truth labeled mask of the query image are supervised at the pixel level using the binary cross-entropy loss function, and the entire network model is backpropagated and updated based on this loss to achieve end-to-end joint optimization of all sub-modules.

2. The method according to claim 1, characterized in that: The preprocessing in step S1 includes uniform scaling to 512×512 resolution, normalizing pixel values ​​to [0,1], and data enhancement through random horizontal flipping, cropping, and color dithering.

3. The method according to claim 2, characterized in that: In step S2, the visual-language model is the CLIP model, which generates semantic prompts based on the PICLIP method, and constructs text prompts in the form of "a photo of {target class}" and "a photo without {target class}".

4. The method according to claim 3, characterized in that: In step S3, the foreground prototype of the supporting image is extracted by masked average pooling, and the matching degree between the query pixel and the foreground prototype is calculated by normalized cosine similarity to generate visual query memory.

5. The method according to claim 4, characterized in that: In step S5, the guided memory refinement module projects multi-source memories onto a unified latent space through a linear layer to construct a query vector. Q Key vector K Value vector V and support key vectors Ks .

6. The method according to claim 5, characterized in that: In step S5, through calculation Q and K The scaled dot product yields the query self-attention matrix. : in Represents matrix multiplication. The dimension of the key vector; the self-attention matrix reflects the autocorrelation of query features in the spatial dimension; combined with the support foreground prototype and Q Cosine similarity constructs foreground enhancement guiding weights .

7. The method according to claim 6, characterized in that: Through formula By fusing attention maps, negative values ​​are preserved to suppress background regions, and a scaling factor α is introduced to adjust the suppression intensity, i.e.: in Represents the normalization operation, symbol This means that only the negative values ​​are retained to suppress the background area. This is a scaling factor to control the intensity of suppression.

8. The method according to claim 7, characterized in that: In step S5, a residual join method is used to add the original semantic query memory to the enhanced query representation, preserving the semantic structure and injecting foreground enhancement information.

9. The method according to claim 8, characterized in that: The binary cross-entropy loss function is used to calculate the pixel-level loss between the predicted segmentation mask and the real labeled mask, thereby enabling backpropagation updates of all network parameters.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the SAM2 few-shot segmentation method based on semantic-visual dual-memory fusion as described in any one of claims 1-9.

Images