Zero-shot learning with dual space prototype calibration and semantic distillation
By constructing a dual-space prototype architecture and a semantic distillation mechanism, the problems of cross-domain offset and external semantic dependence in zero-shot learning are solved, achieving more efficient unseen category recognition and generalization capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIANGTAN UNIV
- Filing Date
- 2026-04-30
- Publication Date
- 2026-07-28
AI Technical Summary
Existing zero-shot learning methods suffer from cross-domain offset problems during semantic calibration and rely too heavily on external semantics during the inference phase, resulting in insufficient generalization ability and high computational cost when facing unseen categories.
We employ a dual-space prototype calibration and semantic distillation approach. During the training phase, we construct a dual prototype architecture consisting of a primary semantic space and a low-rank enhanced semantic space. We then use a semantic distillation mechanism to internalize cross-modal semantic perception capabilities in the visual forward path and perform visual feature extraction and weighted similarity matching with pre-stored prototypes during the inference phase.
It effectively mitigates cross-domain offsets during semantic calibration, reduces computational overhead and memory usage, and improves the model's generalization performance and computational efficiency for unseen categories.
Smart Images
Figure CN122473528A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and artificial intelligence, and relates to a zero-shot learning method for dual-space prototype calibration and semantic distillation. Background Technology
[0002] The development of deep learning technology has enabled breakthroughs in image classification and recognition across numerous fields, largely relying on massive amounts of finely labeled training data. However, in real-world applications, data distribution often exhibits a significant long-tail effect, with new categories constantly emerging, making the labeling of potential image categories extremely costly. This heavy reliance on massive amounts of labeled data limits the generalization and deployment capabilities of traditional supervised learning models in data-scarce scenarios and open environments.
[0003] To address these challenges, Zero-Shot Learning (ZSL) has become crucial. The core motivation of ZSL is to mimic human cognitive reasoning abilities, enabling models to identify unseen classes that have never been seen during training, relying solely on descriptive features of the target concept (such as textual attributes or semantic word vectors). However, when building a semantic bridge for transferring knowledge from visible classes to unseen classes, the inherent modal differences between visual and semantic spaces make it easy for models to extract redundant noise unrelated to the core semantics, such as background, lighting, and occlusion. This makes the model prone to overfitting to these non-essential visual features during training, leading to severe domain bias and generalization decay when facing images of unseen classes with unknown visual distributions.
[0004] In current mainstream technologies, embedding-based methods learn the mapping from visual features to semantic space and perform classification through similarity calculation. For example, ZSLViT, based on the visual Transformer... [1] Enhanced semantically guided visual representation extraction through explicit mining of local block features; a decoupling framework based on attention mechanisms. [2] Cross-attention is used as a feature refiner to decouple independent object attributes and category features from complex visual combinations; semantic neighbor borrowing mechanism. [3] Guided by semantic graphs, local features of visible class instances are adaptively borrowed to improve the construction of the unseen class space; no image classifier injection method is used. [4] A dedicated encoding network was designed to directly utilize text descriptions to generate classification weights for unseen classes in a pre-trained visual model. Although these methods optimize the feature alignment process, they still heavily rely on manually defined semantic information and struggle to fully characterize the highly complex visual features of the real world.
[0005] Generative model-based methods synthesize visual features of unseen classes using generative adversarial networks or diffusion models, transforming zero-shot learning into a fully supervised classification problem. For example, a visually enhanced dynamic semantic prototype mechanism... [5] A visually enhanced dynamic semantic prototype generation strategy is proposed, which effectively alleviates the overfitting of the generator to the visible class; an intelligent synthetic sample selection mechanism is also included. [6] A proximal policy optimization algorithm is introduced, specifically designed to filter and eliminate low-quality generated data in the feature space to ensure feature purity; a semantically guided generative random walk network is also included. [7] By constraining the latent trajectories of the generative model, it can continuously synthesize unseen class samples with high realism and discriminative power; combined with diverse data augmentation methods of the diffusion model. [8] By generating domain-adaptive features, the robustness of the model to test set distribution shifts is improved. However, generative model-based methods are limited by the realism of the synthesized samples and involve large model parameters and complex training and optimization processes, making them computationally expensive and difficult to implement.
[0006] In recent years, with the development of large-scale vision-language pre-trained models such as CLIP [9] With the rise of context-based learning, cue-based learning methods have become a new approach for zero-shot learning. For example, context optimization methods...
[10] It pioneered the introduction of learnable continuous cue vectors in language processing, replacing traditional manual discrete cue templates; and employed a conditional context optimization method.
[11] This introduces a dynamic cueing network, enabling the cue vectors to adaptively adjust based on the input image, thus improving the model's generalization ability on new categories; a multimodal cueing learning framework.
[12] Deep cross-modal collaborative alignment was achieved by symmetrically injecting cue vectors at both the linguistic and visual ends; a self-regulating cueing method was also implemented.
[13] Self-consistent regularization prevents the model from forgetting prior knowledge during the adaptation process. However, existing prompting learning methods often require complex two-branch interactive computations in the inference stage and are highly dependent on external semantic input, which cannot meet the comprehensive requirements of scenarios for computation, semantic input dependence, and high real-time performance.
[0007] In summary, existing zero-shot learning techniques are limited by the coarseness of semantic representation, generative methods are hampered by the enormous training and inference overhead, and large model hinting methods struggle to escape the complex cross-modal computational dependencies. To address this situation, there is an urgent need for a novel collaborative recognition mechanism capable of deeply mining multimodal semantic knowledge and distilling semantic perception capabilities into a single visual forward path during training, thereby improving the model's ability to recognize and generalize to unknown categories.
[0008] [1] Chen S, Hou W, Khan S, et al. Progressive semantic-guided visiontransformer for zero-shot learning. IEEE / CVF Conference on Computer Visionand Pattern Recognition, 2023: 23964-23974.
[0009] [2] Hao S, Han K, Wong K Y K. Learning attention as disentangler forcompositional zero-shot learning. IEEE / CVF Conference on Computer Vision andPattern Recognition, 2023: 15315-15324.
[0010] [3] Hu Y, Zhang L, Wang X, et al. Semantic neighborhood borrowing forzero-shot learning. Proceedings of the AAAI Conference on ArtificialIntelligence, 2024, 38(3): 4567-4575.
[0011] [4] Christensen A, Mancini M, Koepke A S, et al. Image-freeclassifier injection for zero-shot classification. IEEE / CVF InternationalConference on Computer Vision, 2023: 19072-19081.
[0012] [5] Hou M, Liu Y, Zhang Z, et al. Visual augmented dynamic semanticprototype generation for zero-shot learning. IEEE / CVF Conference on ComputerVision and Pattern Recognition, 2024: 15432-15441.
[0013] [6] Gowda S N. Synthetic sample selection for generalized zero-shotlearning. IEEE / CVF Conference on Computer Vision and Pattern RecognitionWorkshops, 2023: 58-67.
[0014] [7] Zhang W, Janson P, Yi K, et al. Continual zero-shot learningthrough semantically guided generative random walks. IEEE / CVF InternationalConference on Computer Vision, 2023: 11574-11585.
[0015] [8] Feng C M, Yu K, Liu Y, et al. Diverse data augmentation withdiffusions for effective test-time prompt tuning. IEEE / CVF InternationalConference on Computer Vision, 2023: 2704-2714.
[0016] [9] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision. International conference on machine learning. PMLR, 2021: 8748-8763.
[0017]
[10] Zhou K, Yang J, Loy C C, et al. Learning to prompt for vision-language models. International Journal of Computer Vision, 2022, 130(9): 2337-2348.
[0018]
[11] Zhou K, Yang J, Loy C C, et al. Conditional prompt learning for vision-language models. IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022: 16816-16825.
[0019]
[12] Khattak M U, Rasheed H, Maaz M, et al. MaPLe: Multi-modal prompt learning. IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023: 19113-19122.
[0020]
[13] Khattak M U, Wasim S T, Naseer M, et al. Self-regulating prompts: Foundational model adaptation without forgetting. IEEE / CVF International Conference on Computer Vision, 2023: 15190-15200. Summary of the Invention
[0021] The purpose of this invention is to address the technical problems existing in current zero-shot learning methods, such as cross-domain semantic calibration offset and excessive dependence on external semantics during the inference stage. This invention proposes a zero-shot learning method based on dual-space prototype calibration and semantic distillation. This method first pre-positions the highly complex semantic interaction and processing to the model training stage, guiding the visual forward path to internalize cross-modal semantic perception capabilities through a semantic distillation mechanism. Then, during the inference stage, the model decouples the computational flow, and the visual forward path extracts visual features through forward propagation, performing weighted similarity matching with pre-stored dual-space unseen class prototypes to complete zero-shot learning classification.
[0022] The technical solution adopted in this invention is
[0023] A zero-shot learning method based on dual-space prototype calibration and semantic distillation is characterized by the following steps: First, a large language model is used to generate deep semantic descriptions of categories. Semantic representations are constructed through constrained refinement and dual filtering, and dual-space prototypes (original and enhanced spaces) are built based on a low-rank matrix. Second, a visual encoder is designed to extract global and local image features. A dual-path gating network consisting of a semantic guidance path and a visual path is designed. Consistency distillation transfers the spatial attention distribution of the semantic guidance path to the visual path. Finally, during the inference phase, the visual path is used. Visual features are extracted forward and weighted similarity matching is performed with the pre-stored dual-space prototypes to complete zero-shot learning classification.
[0024] The beneficial effects of this invention are
[0025] 1. This invention innovatively designs a dual-prototype architecture of a primary semantic space and a low-rank enhanced semantic space, thereby effectively mitigating the cross-domain offset problem during semantic calibration. Specifically, the primary semantic space is responsible for fully preserving the general prior knowledge contained in the pre-trained model; the low-rank enhanced space, relying on low-rank constraints, adaptively completes feature adjustment for the current task. The two complement and synergize with each other, effectively ensuring the model's generalization performance for unseen classes.
[0026] 2. This invention proposes a semantic distillation strategy. First, during the training phase, the mechanism guides the path to actively extract and internalize spatial attention knowledge from semantic branches. Then, during the inference phase, the model achieves effective decoupling of visual and semantic computational flows, generating discriminative attention through visual forward propagation, thereby reducing computational overhead and memory usage.
[0027] 3. This invention ensures cross-class generalization capability through a multi-level constraint mechanism: at the feature level, the visual forward path introduces an information bottleneck layer, forcing the model to generate attention through visual cues, avoiding over-reliance on specific semantic signals; at the semantic level, the low-rank matrix restricts the prototype transformation to a few principal directions, preventing overfitting to the visible class. The two mechanisms work together to exhibit superior generalization performance for both visible and unseen classes. Attached Figure Description
[0028] Figure 1 This is a general framework diagram of the method of the present invention. Detailed Implementation
[0029] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The core of the present invention lies in constructing a zero-shot learning method for dual-space prototype calibration and semantic distillation, which mainly includes three stages: dual-space semantic prototype construction, decoupled dual-path gated visual encoding training, and inference.
[0030] First, a dual-space semantic prototype is constructed. For all category names containing both visible and invisible classes, a large language model is used to generate multiple appearance descriptions for each category. An attribute vocabulary is obtained by automatically mining the training images of the visible classes. Subsequently, the embedded vectors of each description text and the category name are extracted using the frozen CLIP text encoder. Cosine similarity is calculated as a relevance score, and the proportion of attribute vocabulary in each description is calculated as attribute coverage. Combining these two indicators, a high-quality subset of descriptions is selected for each category. The retained description texts are fed into the CLIP text encoder, and the self-attention weights of the last layer are extracted and weighted pooled. The mean of multiple description vectors within the category is taken as the initial prototype. Based on this, a dual prototype space is constructed: the original space prototype directly inherits from the initial prototype and remains fixed, aiming to preserve general semantic priors; the enhanced space prototype is generated from the initial prototype through a learnable low-rank matrix mapping. The matrix is explicitly decomposed into the product of two low-rank matrices, thereby strictly controlling the degrees of freedom of feature transformation and effectively saving storage overhead.
[0031] Secondly, semantic distillation and visual encoding training are performed. The input image, after preprocessing, is fed into a visual Transformer encoder. The final layer classification labels are extracted as global visual features, and the image patch labels are reconstructed according to their spatial location to obtain a spatial feature map. A dual-path gating network is designed. The semantically guided path takes the enhanced spatial prototype as input, passes through two fully connected layers to generate a spatial attention mask, and then performs weighted pooling on the spatial feature map to obtain the semantically guided local enhancement features. The deployment path takes the global visual features as input, and sets a learnable information bottleneck layer before entering the gating network. The output passes through two fully connected layers to generate a spatial attention mask and obtain local enhancement features. The global visual features and the local enhancement features of the two paths are weighted and fused to obtain the final visual representation. The comparative learning loss is calculated simultaneously in the original semantic space and the enhanced semantic space. The mean squared error of the deployment path mask and the semantically guided path mask is calculated as the distillation loss, forcing the deployment path to mimic the attention distribution of the semantically guided path. The semantically guided path mask does not participate in gradient backpropagation.
[0032] Finally, dual-space fusion inference is performed. During inference, computational flow decoupling is implemented. The input image is propagated forward through the visual encoder and the visual forward path to obtain the final visual features. For each unseen class, the cosine similarity between the visual features and the original spatial prototype and the enhanced spatial prototype is calculated separately, and then fused using weighted coefficients. The original spatial prototype is given a higher weight to ensure it does not deviate from the overall structure. The unseen class with the highest comprehensive similarity is selected as the recognition result output.
Claims
1. A zero-shot learning method for layer dual-space prototype calibration and semantic distillation, characterized in that... The process involves constructing dual-space semantic prototypes and refining constrained semantics. First, for both visible and invisible classes, a large language model is used to generate category descriptions. After filtering by relevance and attribute coverage quality, semantic features are extracted to construct a dual-space semantic prototype library containing both original and enhanced spatial prototypes. Next, decoupled dual-path gated visual encoding training is performed to extract image visual features, which are then fed into a dual-path gated network consisting of a semantically guided path and a visual forward path. A contrastive learning loss and a cross-path consistency distillation mechanism are combined to collaboratively optimize the visual encoder and the gated network. Finally, during the inference phase, the visual forward path is used to extract visual features through forward propagation. These features are then matched with the dual-space unseen class prototypes for similarity matching, outputting zero-shot learning classification results.
2. A zero-shot learning method for dual-space prototype calibration and semantic distillation, characterized in that... Consistent distillation of the decoupled dual-path gating network is implemented; global features of the input image are extracted using a visual encoder, and image patch labels are reshaped into spatial feature maps; the global features and spatial feature maps are fed into the decoupled dual-path gating network, where the semantically guided path takes the enhanced spatial prototype as input, generates a spatial attention mask through the network, and performs weighted pooling on the spatial feature map to obtain local enhanced features; The visual forward path takes global features as input, passes through a learnable information bottleneck layer to generate a spatial attention mask and compute local enhancement features; The global features and the local enhancement features output by the dual paths are weighted and fused to generate the final visual features. The contrastive learning loss is calculated simultaneously in the original space and the enhancement space. Finally, the mean square error between the visual forward path mask and the semantic guidance path mask is calculated as the distillation loss to constrain the attention distribution of the visual forward path to fit the semantic guidance path.
3. A zero-shot learning method for dual-space prototype calibration and semantic distillation, characterized in that... Dual-space fusion reasoning; During the inference phase, the cross-modal computation flow is decoupled, the input image completes the forward propagation, and the visual representation is generated from the visual forward path; For each unseen category, the cosine similarity between the obtained visual representation and the original spatial prototype and the enhanced spatial prototype is calculated, and a weighted strategy is used for fusion scoring. Finally, the unseen category with the highest overall similarity score is selected as the recognition result for output.