Small sample semantic segmentation method based on interlayer self-distillation decoupling
By employing a few-sample semantic segmentation method with interlayer self-distillation decoupling, the problems of semantic ambiguity in feature interaction and intra-class differences in existing technologies are solved, improving the segmentation accuracy and interpretability of the model and achieving more efficient target segmentation.
Patent Information
- Application Number
- CN202511503443.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-21
- Publication Date
- 2026-01-16
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing few-sample semantic segmentation methods suffer from interactive semantic ambiguity and intra-class differences during feature interaction, making it difficult for the model to effectively capture common semantics and affecting segmentation performance.
We employ an interlayer self-distillation decoupling method, which combines feature separation within branches and multimodal feature alignment with self-attention and cross-attention to establish distillation loss, thereby enhancing feature representation capabilities and distilling high-level semantics into shallow features.
It improves the model's segmentation accuracy and interpretability for known and unknown targets, reduces semantic interference within query mixed features, and enhances the model's adaptive perception capability.
Smart Images
Figure CN121354110A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision, and particularly relates to a small sample semantic segmentation method based on inter-layer self-distillation decoupling. BACKGROUND
[0002] The small sample semantic segmentation technology aims to use a small amount of pixel-by-pixel labeled samples to guide the mask prediction of unknown targets. The existing small sample semantic segmentation method usually adopts a double-branch network structure of a support branch and a query branch, wherein the support branch contains a small amount of pixel-by-pixel labeled samples (usually 1-5), and the query picture is only one picture to be segmented. The core idea of this method is to learn the segmentation rules from a small amount of labeled support branches, and generalize these rules to similar query pictures. In this method, the small sample semantic segmentation method based on prototype learning has attracted widespread attention from researchers due to its higher interpretability. However, this method mainly establishes coarse-grained visual feature interaction between support foreground and query mixed features, or divides the query mixed features into multiple patches and establishes fine-grained interaction between support foreground and each patch, in order to improve the segmentation performance of the model for unknown targets. In fact, on the one hand, unreasonable feature interaction may cause problems such as semantic ambiguity and semantic bias; on the other hand, due to the inter-class difference between branches, it is difficult for the model to capture the common semantics suitable for the support and query branches at the same time, which is not conducive to improving the segmentation performance of the model. SUMMARY
[0003] To solve the problems of the prior art, the present application provides a small sample semantic segmentation method based on inter-layer self-distillation decoupling. The features within the branch are separated, the multi-modal feature alignment between the branches is established, the expression ability of the features is enhanced, the distillation loss between the multi-modal interaction features is established, and the high-level semantic distillation is guided to the shallow features, so as to improve the adaptive perception ability of the model for known targets in the support picture and unknown targets in the query picture.
[0004] In order to achieve the above purpose, the technical scheme of the present application is as follows: a small sample semantic segmentation method based on inter-layer self-distillation decoupling, comprising the steps of:
[0005] S1, feature encoding, using a group of shared weight visual encoders to map the support picture and the query picture to a visual feature space, and separating the encoded support mixed features into support foreground and background features with the help of the real support mask;
[0006] S2, inter-layer feature separation and interaction, the support foreground feature and the query mixed feature are separated along the channel dimension to establish inter-layer feature gradient, and the self-attention is established in the branch, the cross-attention is established between the branches, the pre-trained text encoder is used to encode the foreground class text label shared between the branches into foreground class text feature, and the multi-modal feature alignment is established between the foreground class text feature and the inter-layer interaction feature, so as to promote the query mixed feature to be decoupled;
[0007] S3, multi-modal interaction feature distillation, a fully connected layer is connected after each layer of multi-modal interaction feature, the corresponding text label is predicted, the cross-entropy loss of the predicted label and the foreground class text label shared between the branches is calculated, and the self-distillation loss is established on the adjacent two layers of features, so as to distill the high layer feature into the shallow layer feature.
[0008] S4, joint guided learning, the support background prototype and the inter-branch interaction prototype are spliced, and the similarity between each prototype and the support mixed feature and the query feature is calculated position by position, the mask of the known target in the support picture and the mask of the unknown target in the query picture are predicted.
[0009] Further, the visual encoder of the set of shared weights in step S1 adopts the VGG-16, ResNet-50 and ResNet-101 networks pre-trained on the ImageNet dataset.
[0010] Further, the inter-layer feature separation and interaction in step S2 comprises the steps of:
[0011] S21, establishing inter-layer feature gradient, the support foreground feature and the query mixed feature are divided into shallow layer feature, intermediate layer feature and high layer feature along the channel dimension, and the channel dimension of the shallow layer feature to the intermediate layer feature and the intermediate layer feature to the high layer feature is expanded by two times, and the length and width of the shallow layer feature to the intermediate layer feature and the intermediate layer feature to the high layer feature are reduced by two times respectively;
[0012] S22, the self-attention is established on the shallow layer feature, the intermediate layer feature and the high layer feature of the double-branch feature respectively, and the cross-attention is established between the self-attention features of the branches, so as to promote the alignment of the visual information between the branches;
[0013] S23, the CLIP-ViT / 32 large language model is used as the text encoder of the foreground class text label shared between the branches, and the multi-modal feature interaction is established between the foreground class text encoding feature and the shallow layer, intermediate layer and high layer interaction feature, so as to guide the decoupling of the query mixed feature by the class text encoding feature, and the multi-modal interaction feature can be represented as:
[0014]
[0015] Wherein, F l tv , and respectively shallow, intermediate and high layer multi-modal interaction features; F l sq 、 and respectively shallow, intermediate and high layer multi-modal interaction features; F t represents foreground class text encoding features shared between branches; d represents feature dimension, and T represents transposition operation.
[0016] Further, the multi-modal interaction feature distillation in step S3 comprises the steps of:
[0017] S31, sending the shallow, intermediate and high layer multi-modal interaction features into a fully connected layer to predict corresponding text labels, and calculating cross-entropy loss between the foreground class text labels shared between branches and the predicted labels respectively;
[0018] S32, calculating the inter-layer loss between the high layer and the shallow layer multi-modal interaction features, guiding the high layer feature to distill to the shallow layer feature, and gradually improving the reliability of the high layer feature while continuously iterating to enhance the expression ability of the shallow layer feature.
[0019] Further, the joint guided learning in step S4 is to calculate the similarity between the support mixed feature, the query mixed feature and each prototype in the prototype set by position, quickly predict the mask of the known target in the support picture and the unknown target in the query picture according to the maximum similarity score, and integrate the cross-entropy loss between layers, the distillation loss, the support loss and the query loss to optimize the model end to end.
[0020] Further, the joint guided learning in step S4 is to evaluate the robustness of the model in processing actual tasks according to the predicted mask of the known target in the support picture, and evaluate the generalization of the model in processing actual tasks according to the predicted mask of the unknown target in the query picture.
[0021] The beneficial effects of the present application are that, compared with the prior art, the present application uses multi-modal feature interaction and inter-branch interaction feature distillation to decouple the query mixed feature, reduces the mutual interference between different semantics in the query mixed feature, improves the segmentation accuracy of the model for known targets and unknown targets, and improves the interpretability of the model. BRIEF DESCRIPTION OF DRAWINGS
[0022] Figure 1 is a principle schematic diagram of a small sample semantic segmentation method based on inter-layer self-distillation decoupling according to the present application;
[0023] Figure 2 is a process schematic diagram of a small sample semantic segmentation method based on inter-layer self-distillation decoupling according to the present application;
[0024] Figure 3 is a segmentation effect schematic diagram of one embodiment of the small sample semantic segmentation method based on inter-layer self-distillation decoupling. DETAILED DESCRIPTION
[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0026] Please refer to Figure 1 and Figure 2 The small sample semantic segmentation method based on inter-layer self-distillation decoupling comprises the following steps:
[0027] S1, feature encoding, using a set of shared weight visual encoders to map the support image and the query image to a visual feature space, and separating the encoded support mixed features into support foreground and background features with the help of the real support mask;
[0028] S2, inter-layer feature separation and interaction, performing inter-layer feature division on the support foreground features and the query mixed features along the channel dimension, establishing an inter-layer feature gradient, establishing self-attention within the branch, establishing cross-attention between the branches, simultaneously encoding the foreground class text labels shared between the branches into foreground class text features using a pre-trained text encoder, and establishing multi-modal feature alignment with the inter-layer interaction features to promote the decoupling of the query mixed features;
[0029] S3, multi-modal interaction feature distillation, connecting a fully connected layer after each layer of multi-modal interaction features, predicting the corresponding text labels, calculating the cross-entropy loss of the predicted labels and the foreground class text labels shared between the branches, and establishing self-distillation loss on the adjacent two layers of features to distill the high-level features into the shallow-level features.
[0030] S4, joint guided learning, splicing the support background prototype and the inter-branch interaction prototype, and calculating the similarity between each prototype and the support mixed features and the query features position by position, predicting the mask of the known target in the support image and the mask of the unknown target in the query image.
[0031] The set of shared weight visual encoders in step S1 adopts the VGG-16, ResNet-50 and ResNet-101 networks pre-trained on the ImageNet dataset.
[0032] The inter-layer feature separation and interaction in step S2 comprises the following steps:
[0033] S21, establishing an inter-layer feature gradient, dividing the support foreground feature and the query mixed feature into shallow layer, middle layer and high layer features along the channel dimension, and expanding the channel dimension of the shallow layer feature to the middle layer feature and the middle layer feature to the high layer feature by two times, and reducing the length and width of the shallow layer feature to the middle layer feature and the middle layer feature to the high layer feature map by two times respectively;
[0034] S22, respectively establishing self-attention on the shallow layer, middle layer and high layer features of the double-branch features, and establishing cross-attention between the inter-branch self-attention features, so as to promote the alignment of the inter-branch visual information;
[0035] S23, using a CLIP-ViT / 32 large language model as a text encoder of the inter-branch shared foreground class text label, and establishing a multi-modal feature interaction between the foreground class text encoding feature and the shallow layer, middle layer and high layer interaction features, so as to guide the decoupling of the query mixed feature by the class text encoding feature, and the multi-modal interaction feature can be represented as:
[0036]
[0037] wherein, F l tv 、 and are respectively the multi-modal interaction features of the shallow layer, the middle layer and the high layer; F l sq 、 and are respectively the shallow layer, the middle layer and the high layer visual interaction features between the double branches; F t represents the foreground class text encoding feature shared between the branches; d represents the feature dimension, and T represents the transposition operation.
[0038] The multi-modal interaction feature distillation in step S3 includes the steps of:
[0039] S31, sending the multi-modal interaction features of the shallow layer, the middle layer and the high layer into a fully connected layer to predict the corresponding text labels, and respectively calculating the cross-entropy loss between the foreground class text label shared between the branches and the predicted label;
[0040] S32, calculating the inter-layer loss between the high layer and the shallow layer multi-modal interaction features, guiding the high layer feature to be distilled to the shallow layer feature, and gradually improving the reliability of the high layer feature while continuously iterating to enhance the expression ability of the shallow layer feature.
[0041] The joint guided learning in the step S4 is to calculate the similarity between the support mixed feature, the query mixed feature and each prototype on the prototype set by position, to quickly predict the mask of the known target in the support picture and the unknown target in the query picture according to the maximum similarity score, and to integrate the cross-entropy loss, the distillation loss, the support loss and the query loss between layers to optimize the model end to end.
[0042] In another embodiment of the present application, the joint guided learning in the step S4 is to evaluate the robustness of the model in processing actual tasks according to the predicted mask of the known target in the support picture, and to evaluate the generalization of the model in processing actual tasks according to the predicted mask of the unknown target in the query picture.
[0043] After the technical solution of the present application is applied, the segmentation effect is as shown in the following table Figure 3 As shown in the following table
[0044] The above is the preferred embodiment of the present application, it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements also as the protection scope of the present application.
Claims
1. A small sample semantic segmentation method based on inter-layer self-distillation decoupling, characterized by, The method comprises the steps of: S1, feature coding, mapping the support image and the query image to a visual feature space by using a set of shared weight visual encoders, and separating the coded support mixed features into support foreground and background features by means of a real support mask; S2, inter-layer feature separation and interaction, performing inter-layer feature division on the support foreground features and the query mixed features along the channel dimension, establishing an inter-layer feature gradient, and establishing self-attention within the branch and cross-attention between the branches, while using a pre-trained text encoder to encode the foreground class text label shared between the branches into foreground class text features, and establishing multi-modal feature alignment with the inter-layer interaction features to promote the decoupling of the query mixed features; S3, multi-modal interaction feature distillation, connecting a fully connected layer after each layer of multi-modal interaction features to predict the corresponding text label, calculating the cross-entropy loss between the predicted label and the foreground class text label shared between the branches, and establishing a self-distillation loss on the adjacent two layers of features to distill the high-level features into the shallow features. S4, joint guided learning, splicing the support background prototype and the inter-branch interaction prototype, and calculating the similarity between each prototype and the support mixed features and the query features position by position to predict the mask of the known target in the support image and the mask of the unknown target in the query image.
2. The small sample semantic segmentation method based on inter-layer self-distillation decoupling according to claim 1, characterized in that, The set of shared weight visual encoders in step S1 adopts the VGG-16, ResNet-50 and ResNet-101 networks pre-trained on the ImageNet dataset.
3. The small sample semantic segmentation method based on inter-layer self-distillation decoupling according to claim 1, wherein, The inter-layer feature separation and interaction in step S2 comprises the steps of: S21, establishing an inter-layer feature gradient, dividing the support foreground features and the query mixed features into shallow, intermediate and high-level features along the channel dimension, and expanding the channel dimension of the shallow features to the intermediate features and the intermediate features to the high-level features by two times, and reducing the length and width of the shallow features to the intermediate features and the intermediate features to the high-level features by two times; S22, establishing self-attention on the shallow, intermediate and high-level features of the double-branch features respectively, and establishing cross-attention between the self-attention features of the branches to promote the alignment of the visual information between the branches; S23, using a CLIP-ViT / 32 large language model as a text encoder of the foreground class text label shared between the branches, and establishing multi-modal feature interaction between the foreground class text encoded features and the shallow, intermediate and high-level interaction features to guide the decoupling of the query mixed features by the class text encoded features, and the multi-modal interaction features can be represented as: where F l tv , and represent the multi-modal interaction features of shallow, intermediate and high layers respectively; F l sq , and represent the visual interaction features of shallow, intermediate and high layers between two branches respectively; F t represents the foreground class text encoding features shared between branches; d represents the feature dimension, and T represents the transpose operation.
4. The small sample semantic segmentation method based on inter-layer self-distillation decoupling according to claim 1, characterized in that, The multi-modal interaction feature distillation in step S3 comprises the steps of: S31, sending the multi-modal interaction features of the shallow, intermediate and high-level features into a fully connected layer to predict the corresponding text label, and calculating the cross-entropy loss between the foreground class text label shared between the branches and the predicted label; S32, calculating the inter-layer loss between the high-level and shallow multi-modal interaction features to guide the distillation of the high-level features to the shallow features, while continuously iterating to enhance the expression ability of the shallow features and gradually improving the reliability of the high-level features.
5. The small sample semantic segmentation method based on inter-layer self-distillation decoupling according to claim 1, wherein, The joint guided learning in step S4 is to calculate the similarity between the support mixed feature, the query mixed feature and each prototype on the prototype set by position, quickly predict the mask of the known target in the support picture and the unknown target in the query picture according to the maximum similarity score, and integrate the cross-entropy loss between layers, the distillation loss, the support loss and the query loss to optimize the model end to end.
6. The small sample semantic segmentation method based on inter-layer self-distillation decoupling according to claim 1, characterized in that, The joint guided learning in step S4 is to evaluate the robustness of the model in processing actual tasks according to the predicted mask of the known target in the support picture and the generalization of the model in processing actual tasks according to the predicted mask of the unknown target in the query picture.