Semantic essence representation large model zero sample learning method

Abstract

The invention discloses a large-model zero-sample learning method for semantic essence representation. The core of the method is a dual-module collaborative optimization framework. Firstly, in a semantic essence representation module, a text encoder encodes a label set and a prompt set, initial category text embedding and unstructured prompt features are generated through dynamic semantic construction, and the features are extracted as dynamic semantic essence anchor points; and then in a visual semantic alignment module, an image encoder extracts visual features of the input image, and establishes mapping between the visual features and text semantics through cross entropy category alignment, Euclidean distance embedding alignment and Zensen-Shannon divergence distribution alignment. The two modules form closed-loop optimization by means of semantic essential anchor points, finally, efficient zero-sample classification is achieved under the condition that data does not need to be labeled, and robustness of semantic changes of prompt words is remarkably improved. The invention aims to provide a zero sample learning method based on a large model so as to solve the problems of projection domain offset, visible class offset, hub point safety, prompt word sensitivity and the like in the existing method. According to the method, cross-class feature extraction of visible classes and invisible classes is enhanced, a prompt design and feature alignment mechanism in a visual language model is promoted to evolve from a static template to a dynamic self-adaption direction, and a key technical thought is provided for performance bottleneck of zero sample learning in an open dynamic environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of zero-shot learning mode classification, and relates to a zero-shot learning method based on a large model. BACKGROUND

[0002] Zero-shot learning, as a frontier direction in the field of machine learning, is committed to transferring the knowledge of visible categories to invisible categories through sharing semantic information, so as to realize the recognition and generalization of invisible categories. In recent years, by excavating the general knowledge and rich semantic representation accumulated in the pre-training process, combined with the research on large language models, multi-modal models and other large models, a more efficient and robust reasoning mechanism has been explored, further expanding the potential of this direction.

[0003] The core of the large model zero-shot learning method is to utilize the powerful generalization ability and deep semantic representation of large-scale pre-training models, so that it can adapt to new tasks or fields only with a small amount of labeled data or even without any new samples. Compared with traditional machine learning and deep learning models which usually rely on a large amount of labeled data for training and need to collect and label data again when facing new fields or tasks, the large model zero-shot learning significantly reduces the cost and time overhead, and provides a feasible path to address the following key challenges: first, in professional fields such as medicine and military where it is difficult to obtain large-scale labeled data, the method can rely on the prior knowledge in the pre-training model for effective inference; second, when the model needs to generalize across domains or tasks, the method realizes efficient domain adaptation through semantic association and knowledge transfer; third, in the face of evolving application requirements, traditional methods often need to be retrained and optimized, while the large model zero-shot learning can significantly shorten the adaptation cycle and support agile deployment.

[0004] The technical breakthrough of large model zero-shot learning began with the exploration of visual and linguistic cross-modal semantic alignment. Among them, CLIP [1] model proposed in 2021 is a landmark achievement. Through the contrastive training of large-scale image-text pairs containing ImageNet [2] and other datasets, the model constructs a shared semantic space of visual and text encoders, realizes the direct mapping of text description image features, and completes the inference of unknown categories without downstream task fine-tuning, greatly expanding the application boundary of zero-shot learning. However, the performance of CLIP is still limited by the semantic deviation of general pre-training data and downstream tasks, and the static alignment mechanism cannot adapt to complex scenarios.

[0005] To alleviate the above problems, researchers focus on data-driven knowledge expansion and generalization enhancement. For example, the ReMap [3] method expands the 1000 classes of ImageNet to 5000 fine-grained subcategories through class semantic remapping, which allows CLIP to perform zero-shot classification on CUB-200[4] Achieving better accuracy on the dataset validates the value of semantic mining of basic data; AZSL [5] Breaking away from the limitations of traditional zero-shot modeling that relies on source-target domain category associations, this approach utilizes 120 million cross-domain unlabeled text-image pairs to construct a semantic-visual mapping dictionary. This allows the model to expand its knowledge base by enriching the semantic information from the pre-training phase. (MSDN) [6] By fusing multimodal data through bidirectional semantic distillation of the source and target models, the feature shift problem caused by data heterogeneity is solved; VP [7] Learnable visual cue blocks are inserted into the input layer of the image encoder to simulate the human cognitive process of focusing on key regions; CPL [8] Abandoning the fixed text templates of visual language models, Maple dynamically generates adaptive prompts through a task-aware condition generator; [9] Learnable tags are inserted into each layer of the visual and text encoder to enable multi-level prompt selection; VPT

[10] By modeling the probability distribution of the prompt space using variational inference, the interference of noisy data on the prompting effect can be reduced.

[0006] [1] Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision[C] / / International Conference on Machine Learning. PMLR, 2021: 8748-8763.

[0007] [2] Deng Jia, Dong Wei, Socher R, et al. ImageNet: A large-scalehierarchical image database[C] / / 2009 IEEE Conference on Computer Vision andPattern Recognition. Miami, FL, USA: IEEE, 2009: 248-255.

[0008] [3] Chen D, Shen Y, Zhang H, et al. Zero-shot logit adjustment[EB / OL]. (2022)[2023-06-20]. arXiv: 2204.11822.

[0009] [4] Welinder P, Branson S, Mita T, et al. Caltech-UCSD Birds 200[R / OL]. California Institute of Technology, 2010.

[0010] [5] Gao R, Wan F, Organisclak D, et al. Absolute zero-shot learning[EB / OL]. (2022) [2023-06-20]. arXiv: 2202.11319.

[0011] [6] Chen S, Hong Z, Xie G S, et al. MSDN: Mutually semanticdistillation network for zero-shot learning[C] / / Proceedings of the 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition. New Orleans, LA,USA: IEEE, 2022: 7612-7621.

[0012] [7] Bahng H, Jahanian A, Sankaranarayanan S, et al. Exploring visualprompts for adapting large-scale models[J / OL]. arXiv preprint, 2022. arXiv:2203.17274.

[0013] [8] Zhou K, Yang J, Loy C C, et al. Conditional prompt learning forvision-language models[C] / / Proceedings of the 2022 IEEE / CVF Conference onComputer Vision and Pattern Recognition. New Orleans, LA, USA: IEEE, 2022:16816-16825.

[0014] [9] Khattak M, Rasheed H, Maaz M, et al. Maple: Multi-modal promptlearning[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition. 2023: 19113-19122.

[0015]

[10] Derakhshani M, Sanchez E, Bulat A, et al. Variational prompttuning improves generalization of vision-language models[J / OL]. arXivpreprint, 2022. Summary of the Invention

[0016] The purpose of this invention is to provide a zero-shot learning method based on a large model to address issues such as projection domain offset, visible class offset, hub point security, and cue word sensitivity in existing methods. This method enhances cross-class feature extraction between visible and invisible classes, driving the evolution of cue design and feature alignment mechanisms in visual language models from static templates to dynamic adaptation, and providing key technical solutions to overcome the performance bottleneck of zero-shot learning in open and dynamic environments.

[0017] The technical solution adopted in this invention is

[0018] The core of this large-scale zero-shot learning method for semantic substance representation is a two-module collaborative optimization framework. First, in the semantic substance representation module, the text encoder encodes the label set and the cue set, generating initial category text embeddings and unstructured cue features through dynamic semantic construction, which are then refined into dynamic semantic substance anchors. Next, in the visual semantic alignment module, the image encoder extracts the visual features of the input image and establishes a mapping between visual features and text semantics through cross-entropy category alignment, Euclidean distance embedding alignment, and Jensen-Shannon divergence distribution alignment. The two modules form a closed-loop optimization through the semantic substance anchors, ultimately achieving efficient zero-shot classification without labeled data and significantly improving robustness to semantic changes in cue words.

[0019] The beneficial effects of this invention are

[0020] 1. Based on a pre-trained model using large-scale visible class data, a semantic essence representation foundation with strong generalization ability is constructed. This step extracts and solidifies the semantic essence prototype of visible classes, forming a reusable prior knowledge base. In the zero-shot reasoning stage, reasoning can be performed using only this prototype and the category semantic description, without needing to re-access the original visible class data. This achieves continuous encapsulation and efficient transfer of semantic knowledge while ensuring data privacy and security.

[0021] 2. Design a representation mechanism based on semantic space alignment to automatically mine reliable visual-semantic correspondences from unlabeled samples of invisible classes. Through a confidence sample selection strategy guided by semantic consistency, construct high-quality semantic essence anchors. This simulates the feature value mining process in unsupervised learning under conditions without any real annotations, thereby improving the model's semantic alignment efficiency and discriminative representation ability for invisible classes at a lower cost.

[0022] 3. An adaptive cue set is introduced to dynamically align and fine-tune the semantic essence prototype. By employing category-level and distribution-level alignment strategies, the model's representational discriminative power in unfamiliar semantic spaces is optimized, enhancing its robustness in recognizing semantically ambiguous and difficult samples. Ultimately, stable and reliable zero-shot generalization from visible to invisible classes is achieved without any real annotations. Attached Figure Description

[0023] Figure 1 This is a diagram illustrating the overall structure of the method of this invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0025] The method is generally divided into three stages: dynamic semantic space construction, visual semantic substance alignment, and distribution consistency-based classification decision.

[0026] (1) The first stage is the construction of the dynamic semantic space. The input includes a basic set of category labels and a set of learnable cue templates. The labels and cue symbols are input into the large model text encoder to generate a set of initial category text embeddings. The core function of the dynamic semantic building module is to model the semantic structural relationships between categories. It does not directly output these initial embeddings, but rather transforms isolated label information into class semantic prototypes with rich connections, thereby forming a structured dynamic semantic space. In this space, the semantic anchor of each category is influenced by its semantic neighbors and contextual cues, thus enabling a more flexible representation of category distributions in different domains.

[0027] (2) The second stage is visual-semantic substance alignment. Visual features are extracted from the query image through a visual encoder. These features are not used directly for matching, but are input into the core semantic substance representation module along with semantic anchors in the dynamic semantic space and corresponding cue features. This module acts as a semantic filter, which guides the visual features to focus on the regions and attributes most relevant to the core concepts of the category defined by the semantic anchors through interactive mechanisms such as cross-modal attention, thereby stripping away irrelevant information such as style and background and outputting a pure image substance representation.

[0028] (3) The third stage is classification decision based on distribution consistency. In this stage, the refined image essence representation is projected into the dynamic semantic space and directly compared with the semantic anchors of each category in the space. Since both are located in the same space rich in semantic relationships, this similarity calculation is essentially an evaluation of the consistency between the image content and the category concept in semantic distribution. Finally, the category with the best semantic match is selected as the result of zero-shot recognition. The entire process effectively improves the model's understanding and generalization ability of the semantics of unknown categories through dynamic anchor construction and essence representation refinement.

Claims

1. A zero-shot learning method for large-scale semantic essence representation models, characterized by... The first step is the construction of a dynamic semantic space. The input includes a basic set of category labels and a set of learnable cue templates. The labels and cue symbols are input into the large model text encoder to generate a set of initial category text embeddings. The core function of the dynamic semantic building module is to model the semantic structural relationships between categories. It does not directly output these initial embeddings, but rather transforms isolated label information into class semantic prototypes with rich connections, thereby forming a structured dynamic semantic space. In this space, the semantic anchor of each category is influenced by its semantic neighbors and contextual cues, thus enabling a more flexible representation of category distributions in different domains.

2. A zero-shot learning method for large-scale semantic essence representation models, characterized by... Next is the visual-semantic substance alignment; the query image extracts visual features through a visual encoder. These features are not used directly for matching, but are input into the core semantic substance representation module along with semantic anchors in the dynamic semantic space and corresponding cue features. This module acts as a semantic filter, which guides the visual features to focus on the regions and attributes most relevant to the core concepts of the category defined by the semantic anchors through interactive mechanisms such as cross-modal attention, thereby stripping away irrelevant information such as style and background and outputting a clean image substance representation.

3. A zero-shot learning method for large-scale semantic essence representation models, characterized by... Finally, a classification decision is made based on distribution consistency. In this stage, the refined image essence representation is projected into a dynamic semantic space and directly compared with the semantic anchors of each category in the space. Since both are located in the same space rich in semantic relationships, this similarity calculation is essentially an evaluation of the consistency between the image content and the category concept in semantic distribution. Finally, the category with the best semantic match is selected as the result of zero-sample recognition.

Images