A few-shot image segmentation method and system based on prompt learning
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUONENG DIGITAL INTELLIGENCE TECHNOLOGY DEVELOPMENT (BEIJING) CO LTD SPACE-TIME INFORMATION BRANCH
- Filing Date
- 2026-02-25
- Publication Date
- 2026-06-26
AI Technical Summary
Existing cue learning methods struggle to capture fine-grained local features in image segmentation tasks, resulting in insufficient segmentation accuracy and stability. This is especially true when there are dynamic scene changes or significant differences between categories, making it difficult to meet the requirements for fine segmentation.
A deep backbone network with shared weights is used for multi-scale feature extraction. Enhanced cue vectors are generated through weighted global average pooling and attention interaction. Combined with cross-scale feature fusion and pixel-level matching, the semantic discrimination ability and segmentation accuracy of the model are improved.
It significantly improves the accuracy and stability of image segmentation, ensuring the precision and spatial continuity of the segmentation results, and maintaining stable segmentation performance in complex scenes.
Smart Images
Figure CN122289676A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a few-shot image segmentation method and system based on cue learning. Background Technology
[0002] Image segmentation, as one of the core tasks of computer vision, is widely used in medical imaging, remote sensing, autonomous driving, and other fields. However, traditional image segmentation methods typically rely on large-scale labeled data, and their performance is prone to degradation when labeling resources are limited or new categories emerge. Few-shot learning aims to train models to effectively predict new categories using a very small number of labeled samples. Existing few-shot image segmentation methods mainly include prototype-based learning and generative model-based methods. Prototype-based learning methods extract category prototypes from the support set and use the similarity between the prototypes and query set features to complete pixel-level classification. These methods achieve certain results under conditions of few samples, but their segmentation accuracy and cross-category generalization ability are limited when dynamic scenes change or there are large differences between categories. Generative model-based methods, such as cue learning, have made significant progress in natural language processing and are gradually being applied to visual tasks. Cue mechanisms guide the model to focus on task-relevant features by introducing learnable or designable prior information, effectively improving the model's generalization ability.
[0003] However, current cue learning is mainly applied to scene-level image classification. For pixel-level dense prediction tasks and pixel-level segmentation tasks, the model is required to capture local fine-grained features. However, existing cue mechanisms focus on global features, which is difficult to meet the requirements of fine segmentation. Furthermore, existing cue learning methods are insufficient in conveying fine-grained semantic information, which makes it difficult for prototype-based matching mechanisms to accurately distinguish local semantic differences, affecting the accuracy and stability of the segmentation results. Summary of the Invention
[0004] In view of this, the present invention proposes a few-shot image segmentation method and system based on cue learning.
[0005] The technical solution of this invention is implemented as follows: The first aspect of this invention provides a few-shot image segmentation method based on cue learning, comprising:
[0006] A deep backbone network with shared weights is used to extract multi-scale features from the support set image and the query set image respectively, so as to obtain support set features and query set features at multiple scales.
[0007] Based on the first mask corresponding to the support set features, the support set features are weighted global average pooling to obtain the image category prototype; and the image category prototype after adaptive transformation is interacted with the preset learnable cue pool through attention interaction, and an enhanced cue vector is generated through residual fusion.
[0008] A priori mask is generated by utilizing the similarity between the query set features, the support set features, and the first mask. The priori masks, the query set features, and the image category prototypes at each scale are then interactively processed and fused across scales to obtain enhanced query set features.
[0009] The similarity between the enhanced query set features and the enhanced cue vector is used to perform pixel-level matching to obtain the target segmentation result.
[0010] Based on the above technical solutions, preferably, the step of performing weighted global average pooling on the support set features based on the first mask corresponding to the support set features to obtain the image category prototype includes:
[0011] Based on the first mask corresponding to each support set feature, a weighted global average pooling is performed on the corresponding support set feature to obtain the image category prototype of each support set feature;
[0012] The average of the multiple image category prototypes is processed to obtain a comprehensive image category prototype, and the comprehensive image category prototype is determined as the image category prototype corresponding to all the support set features.
[0013] Based on the above technical solutions, preferably, the step of performing attention interaction between the adaptively transformed image category prototype and a preset learnable cue pool, and generating an enhanced cue vector through residual fusion, includes:
[0014] Using the image category prototype as the Query and the query set features as the Key, attention interaction is performed through the Query-Key cross-attention mechanism to obtain the intermediate image category prototype;
[0015] The intermediate image category prototype is used as a cue vector, which interacts with a preset learnable cue pool, and an enhanced cue vector is generated through residual fusion.
[0016] Based on the above technical solutions, preferably, the step of generating a priori mask using the similarity between the query set features, the support set features, and the first mask includes:
[0017] The support set features are spatially weighted by using a support set mask to obtain the support set weighted features;
[0018] The similarity between the query set features and the support set weighted features is calculated to obtain the cosine similarity between them.
[0019] The cosine similarity is normalized to generate a priori mask.
[0020] Based on the above technical solutions, preferably, the step of interactively processing and cross-scale fusion of the prior masks, query set features, and image category prototypes at each scale to obtain enhanced query set features includes:
[0021] The prior masks, query set features, and image category prototypes at each scale are interactively processed to obtain interactive information;
[0022] The contextual and detail features of the interaction information are extracted using the void space pyramid pooling module. After adjusting the resolution of the contextual and detail features using bilinear interpolation, feature interaction is performed to obtain multi-scale interaction features.
[0023] The multi-scale interactive features are spliced and fused along the channel dimension to obtain enhanced query set features.
[0024] Based on the above technical solutions, preferably, the step of performing pixel-level matching using the similarity between the enhanced query set features and the enhanced cue vector to obtain the target segmentation result includes:
[0025] Pixel-level matching is performed using the similarity between the enhanced query set features and the enhanced cue vector to obtain a target segmentation mask;
[0026] The target segmentation mask is binarized to obtain the target segmentation result.
[0027] Based on the above technical solutions, preferably, after performing pixel-level matching using the similarity between the enhanced query set features and the enhanced cue vector to obtain the target segmentation result, the method further includes:
[0028] Obtain the loss functions of the target segmentation mask and the real mask, and combine the feature diversity loss and attention sparsity loss to obtain the total training loss;
[0029] The deep backbone network is iteratively trained using the total training loss until the total training loss is less than a preset threshold.
[0030] More preferably, a second aspect of the present invention provides a few-shot image segmentation system based on cue learning, comprising: a feature extraction module, a pooling interaction module, a feature fusion module, and a result prediction module; wherein,
[0031] The feature extraction module is configured to use a deep backbone network with shared weights to perform multi-scale feature extraction on the support set image and the query set image respectively, so as to obtain support set features and query set features at multiple scales.
[0032] The pooling interaction module is configured to perform weighted global average pooling on the support set features based on the first mask corresponding to the support set features to obtain an image category prototype; and to perform attention interaction between the image category prototype after adaptive transformation and a preset learnable cue pool to generate an enhanced cue vector through residual fusion.
[0033] The feature fusion module is configured to generate a priori mask by utilizing the similarity between the query set features, the support set features, and the first mask, and to perform interactive processing and cross-scale fusion of the priori mask, the query set features, and the image category prototype at each scale to obtain enhanced query set features;
[0034] The result prediction module is configured to perform pixel-level matching using the similarity between the enhanced query set features and the enhanced hint vector to obtain the target segmentation result.
[0035] More preferably, a third aspect of the present invention provides an electronic device, including a processor and a memory; the memory has a computer program stored thereon, wherein the computer program, when executed by the processor, implements the few-sample image segmentation method based on cue learning described in the first aspect.
[0036] More preferably, a fourth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the few-sample image segmentation method based on cue learning as described in the first aspect.
[0037] The few-shot image segmentation method and system based on cue learning of the present invention have the following advantages over the prior art:
[0038] 1. A deep backbone network with shared weights is used to extract multi-scale features from the support set and query set images, simultaneously acquiring multi-level feature representations from coarse-grained global structure to fine-grained local texture. Weighted global average pooling is then performed based on the first mask corresponding to the support set features, focusing on target region features and suppressing background interference to generate a cleaner image category prototype. This prototype is then interacted with a pre-defined learnable cue pool through attention, and residual fusion is used to generate enhanced cue vectors, achieving semantic information enhancement from global to local levels and improving the semantic discriminative ability of category prototype matching. Furthermore, a cross-scale feature interaction mechanism is used to integrate spatial correspondences and semantic associations at different scales, enabling the model to more accurately distinguish local semantic differences during pixel-level matching, significantly improving the accuracy and stability of the segmentation results.
[0039] 2. Pixel-level matching is performed by enhancing the similarity between query set features and enhanced suggestion vectors, directly outputting the target segmentation result. This end-to-end pixel-level matching mechanism avoids the spatial information loss caused by feature aggregation in traditional methods, ensuring the precision and spatial continuity of the segmentation result. At the same time, through multi-scale feature fusion and suggestion enhancement mechanisms, the model can maintain stable segmentation performance in complex scenes.
[0040] 3. In the cross-scale feature fusion stage, the cue pool constraint mechanism guides the cue vector to semantically align with the features of the multi-scale query set. This avoids the cue vector from getting stuck in local optima due to sample noise or feature redundancy under conditions of few samples. It ensures that each cue vector focuses on different semantic dimensions, suppresses noise features that are unrelated to the target category, amplifies feature dimensions that are sensitive to local semantic differences, and ensures the accuracy of fine-grained segmentation. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 A flowchart illustrating a few-shot image segmentation method based on cue learning provided in an embodiment of the present invention;
[0043] Figure 2 A schematic diagram of the overall framework of a few-shot image segmentation method based on cue learning provided in an embodiment of the present invention;
[0044] Figure 3 A schematic diagram of the prompt enhancement module framework provided in an embodiment of the present invention;
[0045] Figure 4 This is a schematic diagram of the multi-scale feature interaction enhancement module framework provided in an embodiment of the present invention;
[0046] Figure 5 Provided for embodiments of the present invention A schematic diagram showing the experimental results comparing 1-shot and 5-shot segmentation on the dataset;
[0047] Figure 6 Provided for embodiments of the present invention A schematic diagram showing the experimental results comparing 1-shot and 5-shot segmentation on the dataset;
[0048] Figure 7This is a schematic diagram of the structure of a few-shot image segmentation system based on cue learning provided in an embodiment of the present invention;
[0049] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0050] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0051] In some embodiments, such as Figure 1 and Figure 2 As shown, Figure 1 A flowchart illustrating a few-shot image segmentation method based on cue learning provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall framework of a few-shot image segmentation method based on cue learning provided in an embodiment of the present invention; the few-shot image segmentation method based on cue learning provided by the present invention includes:
[0052] S110 utilizes a deep backbone network with shared weights to perform multi-scale feature extraction on the support set image and query set image respectively, obtaining support set features and query set features at multiple scales.
[0053] Here, the deep backbone network can be ResNet50 or ResNet101. Using a backbone network with shared weights to extract features from the support set images and query set images separately, obtaining multi-scale features, can ensure the consistency of the feature extraction process.
[0054] In an alternative embodiment, with For example, its 20 categories are evenly divided into 4 folds, with each fold containing 5 validation categories; for The dataset is divided into 4 folds, with each fold containing 20 validation classes. During the partitioning process, the training and validation classes are ensured to be mutually exclusive to effectively evaluate the model's cross-class generalization ability. Each training task consists of a query image and K support samples of the same class. Support set samples are obtained through random sampling without replacement to improve sample diversity and reduce the risk of overfitting. For 1-shot tasks, each support set contains only a single sample; for 5-shot tasks, each support set contains 5 independent samples. Specifically, the support set samples... ,in, Let be the i-th support sample. Each support sample contains an image and its corresponding segmentation mask.
[0055] All input images are uniformly resized to a fixed size, such as 473×473 pixels, and standardized, including operations such as subtracting the mean and dividing by the standard deviation. The segmentation mask is converted into a binary mask, where the pixel value of the foreground target region is 1, and the pixel value of the background region is 0.
[0056] For the support set image s_i and the query image q, features are extracted through the backbone network F_backbone:
[0057]
[0058] in, This represents the feature map of the i-th supporting sample. This represents the feature map of the query image. The extracted feature map contains multiple scales, denoted as... , These correspond to the feature maps output by different levels of the backbone network.
[0059] S120: Based on the first mask corresponding to the support set features, perform weighted global average pooling on the support set features to obtain the image category prototype; and perform attention interaction between the adaptively transformed image category prototype and the preset learnable cue pool to generate an enhanced cue vector through residual fusion.
[0060] Here, the first mask can be generated by labeling the support set image or by a weakly supervised method, and is used to label the target region and the background region. Feature weighting is achieved by multiplying the support set features pixel-by-pixel with the first mask, preserving the target region features while suppressing the background region. Global average pooling is then performed to generate an image category prototype vector. This prototype only contains the average features of the target region, avoiding background noise interference and improving prototype purity.
[0061] In some embodiments, weighted global average pooling is performed on the support set features based on a first mask corresponding to the support set features to obtain an image category prototype, including:
[0062] Based on the first mask corresponding to each support set feature, weighted global average pooling is performed on the corresponding support set features to obtain the image category prototype of each support set feature;
[0063] The average of multiple image category prototypes is processed to obtain a comprehensive image category prototype, which is then determined as the image category prototype corresponding to all support set features.
[0064] For a single-sample support set image (1-shot), the class prototype P is calculated as follows:
[0065]
[0066] in, Indicates support set features, This represents the first mask corresponding to the features of the support set. This represents the summation operation for all pixels within the mask area.
[0067] For a multi-sample support set image (K-shot), the class prototype P is calculated as follows:
[0068]
[0069] in, This represents the prototype of the image category generated by the i-th support set feature.
[0070] In some embodiments, the adaptively transformed image category prototype is subjected to attentional interaction with a preset learnable cue pool, and an enhanced cue vector is generated through residual fusion, including:
[0071] The image category prototype is used as the query, and the query set features are used as the key. Attention interaction is performed through the query-key cross attention mechanism to obtain the intermediate image category prototype.
[0072] The intermediate image category prototype is used as a cue vector, which interacts with a preset learnable cue pool, and an enhanced cue vector is generated through residual fusion.
[0073] In this embodiment, please refer to Figure 3 , Figure 3 This is a schematic diagram of the prompt enhancement module framework provided in an embodiment of the present invention; the image category prototype P is used as the query Query, and the query set features are... As the key, the feature adaptive transformation is achieved through the query-key cross-attention mechanism:
[0074]
[0075] in, Let be the learnable projection matrix, d be the feature dimension, and A be the attention weight matrix. This is a prototype for an intermediate image category.
[0076] intermediate image category prototype As a cue vector, it is compared with the preset learnable cue pool. Interact:
[0077]
[0078] in, and For a learnable projection matrix, To indicate attention weights, To enhance the cue vector.
[0079] S130 generates a priori mask by utilizing the similarity between query set features, support set features, and the first mask. The priori masks, query set features, and image category prototypes at each scale are then interactively processed and fused across scales to obtain enhanced query set features.
[0080] Prior masks can accurately locate target regions in the query set through similarity calculations, avoiding the loss of local features caused by global average pooling. Multi-scale feature interaction dynamically adjusts feature weights through an attention mechanism, enabling the model to distinguish local differences. Cross-scale fusion and prototype interaction preserve original feature information through residual connections, avoiding feature degradation due to insufficient sample size.
[0081] In some embodiments, a priori mask is generated using the similarity between query set features, support set features, and a first mask, including:
[0082] The support set features are spatially weighted by using a support set mask to obtain the support set weighted features.
[0083] The similarity between the query set features and the support set weighted features is calculated to obtain the cosine similarity between them.
[0084] The cosine similarity is normalized to generate a priori mask.
[0085] In this embodiment, query set features are calculated. With support set features and corresponding masks Similarity between them, generating a priori masks To highlight key semantic information and suppress background interference:
[0086]
[0087] in, This indicates the calculation of cosine similarity, and Normalize indicates the normalization operation, which makes the mask value range between [0, 1].
[0088] In some embodiments, prior masks, query set features, and image category prototypes at various scales are interactively processed and fused across scales to obtain enhanced query set features, including:
[0089] The prior masks, query set features, and image category prototypes at various scales are interactively processed to obtain interactive information;
[0090] The contextual and detail features of the interaction information are extracted using the hollow space pyramid pooling module. After adjusting the resolution of the contextual and detail features using bilinear interpolation, feature interaction is performed to obtain multi-scale interaction features.
[0091] Multi-scale interactive features are spliced and fused along the channel dimension to obtain enhanced query set features.
[0092] Please see Figure 4 , Figure 4 This is a schematic diagram of the multi-scale feature interaction enhancement module framework provided in an embodiment of the present invention. Taking the dilated spatial pyramid pooling module ASPP as an example, the ASPP module includes four parallel branches: a 1×1 convolution branch, two 3×3 deformable convolution branches with dilation rates of 6 and 12 respectively, and a 3×3 standard dilated convolution branch. The implementation of each branch in the ASPP module is as follows:
[0093] Branch_1=Conv_1×1(Input)→BN→ReLU→Dropout(0.1)
[0094] Branch_2=DeformConv_3×3_rate6(Input)→BN→ReLU→Dropout(0.1)
[0095] Branch_3=DeformConv_3×3_rate12(Input)→BN→ReLU→Dropout(0.1)
[0096] Branch_4=AtrousConv_3×3(Input)→BN→ReLU→Dropout(0.1)
[0097] Where Conv represents convolution operation, DeformConv represents deformable convolution operation, AtrousConv represents dilated convolution operation, BN represents batch normalization, ReLU represents modified linear unit activation function, and Dropout represents random deactivation operation.
[0098] The outputs of each branch are concatenated and then their dimensions are adjusted using a 1×1 convolution.
[0099] Output_ASPP =Conv_1×1(Concat[Branch_1,Branch_2,Branch_3,Branch_4])→BN→ReLU
[0100] Here, the feature dimension output by the ASPP module is C, for example, C=64.
[0101] Prior mask Query set characteristics The image category prototype P is concatenated and input into the multi-scale feature interaction module for processing to obtain interaction information:
[0102]
[0103] right Using the ASPP module to extract multi-scale contextual and detailed features:
[0104]
[0105] Bilinear interpolation is used to adjust the resolution of features at different scales, and feature interaction is performed:
[0106]
[0107] in, Represents the features at the l-th scale. This indicates a bilinear interpolation operation.
[0108] The features processed at each scale are concatenated along the channel dimension:
[0109]
[0110] in, This is for the final enhanced query set features.
[0111] S140, pixel-level matching is performed using the similarity between the enhanced query set features and the enhanced cue vector to obtain the target segmentation result.
[0112] Semantic alignment can be achieved by calculating similarity between enhanced query set features and enhanced suggestion vectors in the feature space. Cosine similarity or Euclidean distance is used to measure the degree of matching between pixel-level feature vectors and suggestion vectors; higher similarity indicates a greater probability that the pixel belongs to the target category.
[0113] In some embodiments, pixel-level matching is performed using the similarity between enhanced query set features and enhanced cue vectors to obtain target segmentation results, including:
[0114] Pixel-level matching is performed using the similarity between enhanced query set features and enhanced cue vectors to obtain a target segmentation mask;
[0115] The target segmentation mask is binarized to obtain the target segmentation result.
[0116] In this embodiment, enhanced cue vectors are used. Enhanced query set features Pixel-level matching is performed based on the similarity between them:
[0117]
[0118] in, A target segmentation mask is used, and the sigmoid function is applied to map similarity scores to the [0, 1] interval. Finally, the... Binarization yields the target segmentation result.
[0119] In some embodiments, after performing pixel-level matching using the similarity between enhanced query set features and enhanced cue vectors to obtain the target segmentation result, the method further includes:
[0120] Obtain the loss functions of the target segmentation mask and the real mask, and combine the feature diversity loss and attention sparsity loss to obtain the total training loss;
[0121] The deep backbone network is iteratively trained using the total training loss until the total training loss is less than a preset threshold.
[0122] In this embodiment, a meta-learning training strategy based on a prototype network is employed, dividing categories into base classes and new classes. Each training epoch involves random sampling of queries from the base classes and K-shot support sets for end-to-end training. For each training epoch, a base class c is randomly selected, and K support samples are randomly selected from that class. A training task is formed by a support sample and a query sample q. The model predicts the segmentation mask of the target in the query sample based on the support samples and calculates the loss by comparing it with the real mask.
[0123] Dice weighted loss was used during training. To address class imbalance:
[0124]
[0125] in, Segment the mask for the target. For a realistic mask, This is a smoothing term used to avoid cases where the denominator is zero.
[0126] Simultaneously, feature diversity loss is introduced. and attention sparsity loss :
[0127]
[0128]
[0129]
[0130]
[0131] Where S represents the similarity matrix of the cue vectors, This represents the similarity matrix after removing elements from the diagonal. This represents the attention weight of the nth feature in batch b. This represents the entropy value of this batch.
[0132] The total loss function L is:
[0133]
[0134] in, and These are weighting coefficients used to balance the contributions of various losses.
[0135] exist and On datasets such as [dataset name missing], 1-shot and 5-shot segmentation experiments were conducted to compare and analyze segmentation accuracy and generalization ability. Evaluation metrics included mean intersection-over-union (mIoU) and binary F1 score (FB). Relevant experimental results are as follows: Figure 5 and Figure 6 As shown.
[0136] Experimental results show that the method of the present invention, under 1-shot conditions, The average mIoU on the dataset reached 62.8%, an improvement of 5.3 percentage points compared to the baseline method; under 5-shot conditions, the average mIoU reached 65.7%, an improvement of 6.1 percentage points compared to the baseline method. In more challenging... On the dataset, it also shows a clear advantage, with an average mIoU of 43.6% under 1-shot conditions and 47.2% under 5-shot conditions.
[0137] In some embodiments, please refer to Figure 7 , Figure 7 This is a schematic diagram of a few-shot image segmentation system based on cue learning, provided in an embodiment of the present invention. The present invention provides a few-shot image segmentation system 700 based on cue learning, comprising: a feature extraction module 710, a pooling interaction module 720, a feature fusion module 730, and a result prediction module 740; wherein,
[0138] The feature extraction module 710 is configured to use a deep backbone network with shared weights to perform multi-scale feature extraction on the support set image and the query set image respectively, so as to obtain support set features and query set features at multiple scales.
[0139] The pooling interaction module 720 is configured to perform weighted global average pooling on the support set features based on the first mask corresponding to the support set features to obtain the image category prototype; and to perform attention interaction between the adaptively transformed image category prototype and the preset learnable cue pool to generate an enhanced cue vector through residual fusion.
[0140] The feature fusion module 730 is configured to generate a priori mask by utilizing the similarity between query set features, support set features and the first mask, and to perform interactive processing and cross-scale fusion of prior masks, query set features and image category prototypes at each scale to obtain enhanced query set features;
[0141] The result prediction module 740 is configured to perform pixel-level matching using the similarity between the enhanced query set features and the enhanced cue vector to obtain the target segmentation result.
[0142] In some embodiments, the pooling interaction module 720 is specifically configured as follows:
[0143] Based on the first mask corresponding to each support set feature, weighted global average pooling is performed on the corresponding support set features to obtain the image category prototype of each support set feature;
[0144] The average of multiple image category prototypes is processed to obtain a comprehensive image category prototype, which is then determined as the image category prototype corresponding to all support set features.
[0145] In some embodiments, the pooling interaction module 720 is specifically configured as follows:
[0146] The image category prototype is used as the query, and the query set features are used as the key. Attention interaction is performed through the query-key cross attention mechanism to obtain the intermediate image category prototype.
[0147] The intermediate image category prototype is used as a cue vector, which interacts with a preset learnable cue pool, and an enhanced cue vector is generated through residual fusion.
[0148] In some embodiments, the feature fusion module 730 is specifically configured as follows:
[0149] The support set features are spatially weighted by using a support set mask to obtain the support set weighted features.
[0150] The similarity between the query set features and the support set weighted features is calculated to obtain the cosine similarity between them.
[0151] The cosine similarity is normalized to generate a priori mask.
[0152] In some embodiments, the feature fusion module 730 is specifically configured as follows:
[0153] The prior masks, query set features, and image category prototypes at various scales are interactively processed to obtain interactive information;
[0154] The contextual and detail features of the interaction information are extracted using the hollow space pyramid pooling module. After adjusting the resolution of the contextual and detail features using bilinear interpolation, feature interaction is performed to obtain multi-scale interaction features.
[0155] Multi-scale interactive features are spliced and fused along the channel dimension to obtain enhanced query set features.
[0156] In some embodiments, the result prediction module 740 is specifically configured as follows:
[0157] Pixel-level matching is performed using the similarity between enhanced query set features and enhanced cue vectors to obtain a target segmentation mask;
[0158] The target segmentation mask is binarized to obtain the target segmentation result.
[0159] In some embodiments, the few-shot image segmentation system based on cue learning further includes an iterative training module; the iterative training module is configured as follows:
[0160] Obtain the loss functions of the target segmentation mask and the real mask, and combine the feature diversity loss and attention sparsity loss to obtain the total training loss;
[0161] The deep backbone network is iteratively trained using the total training loss until the total training loss is less than a preset threshold.
[0162] It should be noted that the few-shot image segmentation system based on cue learning provided in this application embodiment and the few-shot image segmentation method based on cue learning provided in this application embodiment are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned few-shot image segmentation method based on cue learning, and the repeated parts will not be described again.
[0163] In some embodiments, please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 800 provided in this application includes a processor 810 and a memory 820; the memory 820 stores a computer program, wherein the computer program, when executed by the processor, implements the aforementioned few-sample image segmentation method based on cueing learning.
[0164] Specifically, processor 810 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 810 may also include onboard memory for caching purposes. Processor 810 may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.
[0165] Memory 820 may be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory 820 may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory 820 include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and may also be random access memory (RAM) or flash memory; and / or wired / wireless communication links.
[0166] This application also provides a computer-readable medium storing a computer program that, when executed by a processor, implements the aforementioned few-shot image segmentation method based on cue learning. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.
[0167] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.
[0168] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.
Claims
1. A few-shot image segmentation method based on cue learning, characterized in that, include: A deep backbone network with shared weights is used to extract multi-scale features from the support set image and the query set image respectively, so as to obtain support set features and query set features at multiple scales. Based on the first mask corresponding to the support set features, a weighted global average pooling is performed on the support set features to obtain the image category prototype; The image category prototype after adaptive transformation is then interacted with a preset learnable cue pool through attentional interaction, and an enhanced cue vector is generated through residual fusion. A priori mask is generated by utilizing the similarity between the query set features, the support set features, and the first mask. The priori masks, the query set features, and the image category prototypes at each scale are then interactively processed and fused across scales to obtain enhanced query set features. The similarity between the enhanced query set features and the enhanced cue vector is used to perform pixel-level matching to obtain the target segmentation result.
2. The few-shot image segmentation method based on cue learning as described in claim 1, characterized in that, The step of performing weighted global average pooling on the support set features based on the first mask corresponding to the support set features to obtain the image category prototype includes: Based on the first mask corresponding to each support set feature, a weighted global average pooling is performed on the corresponding support set feature to obtain the image category prototype of each support set feature; The average of the multiple image category prototypes is processed to obtain a comprehensive image category prototype, and the comprehensive image category prototype is determined as the image category prototype corresponding to all the support set features.
3. The few-shot image segmentation method based on cue learning as described in claim 1, characterized in that, The step of interacting with the adaptively transformed image category prototype and a preset learnable cue pool through attention, and generating an enhanced cue vector through residual fusion, includes: Using the image category prototype as the Query and the query set features as the Key, attention interaction is performed through the Query-Key cross-attention mechanism to obtain the intermediate image category prototype; The intermediate image category prototype is used as a cue vector, which interacts with a preset learnable cue pool, and an enhanced cue vector is generated through residual fusion.
4. The few-shot image segmentation method based on cue learning as described in claim 1, characterized in that, The step of generating a priori mask using the similarity between the query set features, the support set features, and the first mask includes: The support set features are spatially weighted by using a support set mask to obtain the support set weighted features; The similarity between the query set features and the support set weighted features is calculated to obtain the cosine similarity between them. The cosine similarity is normalized to generate a priori mask.
5. The few-shot image segmentation method based on cue learning as described in claim 1, characterized in that, The step of interactively processing and cross-scale fusion of the prior masks at each scale, the query set features, and the image category prototypes to obtain enhanced query set features includes: The prior masks, query set features, and image category prototypes at each scale are interactively processed to obtain interactive information; The contextual and detail features of the interaction information are extracted using the void space pyramid pooling module. After adjusting the resolution of the contextual and detail features using bilinear interpolation, feature interaction is performed to obtain multi-scale interaction features. The multi-scale interactive features are spliced and fused along the channel dimension to obtain enhanced query set features.
6. The few-shot image segmentation method based on cue learning as described in claim 1, characterized in that, The step of performing pixel-level matching based on the similarity between the enhanced query set features and the enhanced cue vector to obtain the target segmentation result includes: Pixel-level matching is performed using the similarity between the enhanced query set features and the enhanced cue vector to obtain a target segmentation mask; The target segmentation mask is binarized to obtain the target segmentation result.
7. The few-shot image segmentation method based on cue learning as described in claim 6, characterized in that, After performing pixel-level matching using the similarity between the enhanced query set features and the enhanced cue vector to obtain the target segmentation result, the process further includes: Obtain the loss functions of the target segmentation mask and the real mask, and combine the feature diversity loss and attention sparsity loss to obtain the total training loss; The deep backbone network is iteratively trained using the total training loss until the total training loss is less than a preset threshold.
8. A few-shot image segmentation system based on cue learning, characterized in that, include: The module comprises a feature extraction module, a pooling interaction module, a feature fusion module, and a result prediction module; among which, The feature extraction module is configured to use a deep backbone network with shared weights to perform multi-scale feature extraction on the support set image and the query set image respectively, so as to obtain support set features and query set features at multiple scales. The pooling interaction module is configured to perform weighted global average pooling on the support set features based on the first mask corresponding to the support set features to obtain an image category prototype; and to perform attention interaction between the image category prototype after adaptive transformation and a preset learnable cue pool to generate an enhanced cue vector through residual fusion. The feature fusion module is configured to generate a priori mask by utilizing the similarity between the query set features, the support set features, and the first mask, and to perform interactive processing and cross-scale fusion of the priori mask, the query set features, and the image category prototype at each scale to obtain enhanced query set features; The result prediction module is configured to perform pixel-level matching using the similarity between the enhanced query set features and the enhanced hint vector to obtain the target segmentation result.
9. An electronic device comprising a processor and a memory; said memory having a storage for a computer program, wherein, When the computer program is executed by the processor, it implements the few-shot image segmentation method based on cue learning as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the few-shot image segmentation method based on cue learning as described in any one of claims 1 to 7.