Visual language zero-shot learning method for opening visual attributes

Abstract

This invention presents a zero-shot learning method for visual language based on open visual attributes. It automatically constructs an attribute dictionary, improves the model's generalization ability, and promotes zero-shot learning towards an open world. The method is characterized by first constructing an open attribute dictionary using a visual language model; then generating dynamic attribute vectors using the attribute dictionary; and finally, interacting and aligning the visual features extracted by the CLIP image encoder with the dynamically generated attribute vectors through attention, achieving visual localization of the attributes. The two-path features are then aligned and fused, and input into a classifier to obtain the final prediction. The purpose of this invention is to provide a zero-shot learning method for visual language based on open visual attributes, solving the problems of existing zero-shot learning methods such as reliance on predefined, closed, and fixed attribute sets, poor scalability, and semantic gaps. This enhances the applicability of zero-shot learning, improves its scalability across different datasets, and increases the model's generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of zero-shot learning classification technology, and relates to a zero-shot learning method for visual language with open visual attributes. Background Technology

[0002] Transfer learning is a machine learning method [1] The core idea is to apply knowledge and model parameters learned from one task (source domain) to another related but different new task (target domain) by adjusting and reusing them. The typical process involves first pre-training the model in the source domain to learn common low-level features. Then, using this pre-trained model as the starting point for the target task, it is fine-tuned using specific data from the target domain, i.e., adjusting some or all network layer parameters to adapt to the new task. [2] The core advantage of this method lies in its ability to overcome the bottleneck of insufficient data or limited computing resources for the target task, significantly improving the model's performance and training efficiency in small-sample scenarios, and effectively preventing overfitting. It has become a core paradigm in fields such as computer vision and natural language processing, and is a crucial technology for promoting the widespread application of artificial intelligence.

[0003] Zero-shot learning [3] Zero-shot learning is an advanced extension of transfer learning in situations with extremely limited data. Its core goal is to enable the model to recognize or understand entirely new categories that have never been seen in any labeled samples during the training phase. Unlike fine-tuning in transfer learning, the key to zero-shot learning lies in establishing a universal mapping bridge from the "visual feature space" to the "semantic attribute space." Specifically, the model is first trained on images of visible categories and their rich semantic descriptions, learning the association between visual features and high-level semantic concepts. [4] During testing, when faced with an unseen category, the model does not directly identify its pixel patterns. Instead, it maps the extracted visual features to the same semantic space using its pre-known semantic description (e.g., "has black and white stripes, looks like a horse, lives on grasslands"), and infers the best-matching category label by calculating semantic similarity. Therefore, it relies on "knowledge transfer," that is, using semantic-visual relationships learned from visible categories, and generalizing to unseen categories with similar descriptions through shared semantic descriptions as an intermediary. [5] .

[0004] Zero-shot learning methods are mainly divided into two categories: attribute / semantic embedding-based methods and generative model-based methods. Attribute / semantic embedding-based methods... [6][7]The approach relies on a shared "semantic space" as a bridge. This space is typically composed of manually labeled attribute vectors or word vectors learned from large-scale text. During training, the model learns to map the visual features of an image to this semantic space. During prediction, for images with unseen categories, the model maps their visual features to the same semantic space and then classifies them by calculating the similarity between the image and the semantic description vectors of all unseen categories; this is based on a generative model. [8][9] The method uses generative models such as generative adversarial networks or variational autoencoders to synthesize virtual visual feature samples of unseen categories using semantic descriptions of those categories. Once these virtual features are generated, the zero-shot learning problem is transformed into a standard supervised classification problem, where any classifier can be directly used to train and classify the synthesized unseen category features together with the real seen category features.

[0005] In recent years, the emergence of large-scale visual language models has brought about a paradigm shift in the field of computer vision. These models have achieved powerful open-ended visual concept understanding and zero-shot transfer capabilities through comparative learning on ultra-large-scale image-text pairs. Therefore, the more advanced zero-shot learning methods have incorporated the assistance of large models to improve model performance. For example, Reference

[10] combines the general knowledge of CLIP with the fine-grained attribute perception capability of traditional ZSL through attribute hint tuning and cross-network feature alignment; Reference

[11] proposes a method to generate category-level text descriptions using large language models and fine-tunes visual language models to improve their performance in fine-grained zero-shot classification tasks. Reference

[12] proposes a zero-shot classification method called ZLaP, which uses the graph structure of unlabeled data to improve the classification performance of visual language models through label propagation.

[0006] Existing zero-shot learning suffers from problems such as reliance on predefined, closed, and fixed attribute sets, poor scalability, and semantic gaps. To address these issues, this patent proposes a zero-shot visual language learning method based on open visual attributes. First, an open attribute dictionary is generated using a large model. Then, dynamic attribute vectors are obtained from the attribute dictionary and interacted with and attention-aligned with visual features obtained from the CLIP image encoder to achieve visual localization of attributes. Finally, the localized visual features are aligned and fused with the final layer features of a lightweight multimodal large model through dual-path feature alignment and fusion, and a classifier is used to obtain the final prediction. This solves the problems of zero-shot learning, including reliance on predefined, closed, and fixed attribute sets, poor scalability, and semantic gaps.

[0007] [1]S. J. Pan, Q. Yang. A survey on transfer learning. IEEETransactions On Knowledge Data Engineering. 2010, 22(10):1345-1359.

[0008] [2]L. Zhang, X. Gao. Transfer adaptation learning: a decade survey.IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(1):23-44.

[0009] [3]S. Rahman, S. Khan and N. Barnes, Deep0Tag: Deep Multiple InstanceLearning for Zero-Shot Image Tagging. IEEE Transactions on Multimedia,2020,22(1):242-255.

[0010] [4]X. Han, B. Singh, V. I. Morariu and L. S. Davis. VRFP: On-the-FlyVideo Retrieval Using Web Images and Fast Fisher Vector Products. IEEETransactions on Multimedia, vol. 19, no. 7, pp. 1583-1595, 2017.

[0011] [5]Z. Wang, R. Hu, C. Liang; Y. Yu; J. Jiang; M. Ye; J. Chen; Q.Leng. Zero-Shot Person Re-identification via Cross-View Consistency. IEEETransactions on Multimedia, 2016, 18(2):260-272.

[0012] [6]L. Chen, H. Zhang, J. Xiao, W. Liu and S. -F. Chang, “Zero-ShotVisual Recognition Using Semantics-Preserving Adversarial EmbeddingNetworks,” 2018 IEEE / CVF Conference on Computer Vision and PatternRecognition (CVPR), 2018, pp. 1043-1052.

[0013] [7]L. Zhang, T. Xiang and S. Gong, “Learning a Deep Embedding Modelfor Zero-Shot Learning,” 2017 IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2017, pp. 3010-3019.

[0014] [8]J. Liu, H. Bai, H. Zhang, and L. Liu, “Near-Real FeatureGenerative Network for Generalized Zero-Shot Learning,” in 2021 IEEEInternational Conference on Multimedia and Expo (ICME), Jul. 2021, pp. 1-6.

[0015] [9]J. Guan, Z. Lu, T. Xiang, A. Li, A. Zhao and J. -R. Wen. Zero andFew Shot Learning with Semantic Feature Synthesis and Competitive Learning.EEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(7):2510-2523.

[0016]

[10] Xie, G. Li, J. Guo, T. Shu, X. Zhao, F. Zhang, Z. Shao, Ling.Attribute Prompt Alignment Network for Zero-Shot Learning. IEEE Transactionson Neural Networks and Learning Systems, 2025, 36(12): 20457-20463.

[0017]

[11] O. Saha, G. Van Horn and S. Maji, "Improved Zero-ShotClassification by Adapting VLMs with Text Descriptions," 2024 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 17542-17552.

[0018]

[12] V. Stojnić, Y. Kalantidis and G. Tolias, "Label Propagation for Zero-shot Classification with Vision-Language Models," 2024 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 23209-23218. Summary of the Invention

[0019] The purpose of this invention is to provide a zero-shot visual language learning method with open visual attributes, which solves the problems of existing zero-shot learning that rely on predefined, closed and fixed attribute sets, have poor scalability and semantic gap, enhance the applicability of zero-shot learning, improve the scalability of zero-shot learning to different datasets, and increase the generalization ability of the model.

[0020] The technical solution adopted in this invention is

[0021] A zero-shot visual language learning method with open visual attributes automatically constructs an attribute dictionary, improves the generalization ability of the model, and promotes zero-shot learning to the open world. Its key features include: first, constructing an open attribute dictionary using a visual language model; then, generating dynamic attribute vectors using the attribute dictionary; finally, using a CLIP image encoder to extract visual features and interact with and align the dynamically generated attribute vectors to achieve visual localization of the attributes; aligning and fusing the dual-path features; and inputting them into a classifier to obtain the final prediction.

[0022] The beneficial effects of this invention are

[0023] 1. An open-ended attribute discovery method was designed, which solves the dependence of zero-shot learning on predefined attribute sets, enabling the model to adapt to more open and complex real-world scenarios.

[0024] 2. Construct a heterogeneous architecture that allows fine-grained perception pathways and semantic deep reasoning pathways to run in parallel, ensuring that local visual evidence and global semantic reasoning mutually verify and complement each other. Attached Figure Description

[0025] Figure 1 This is a diagram illustrating the overall structure of the method of this invention. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0027] The method is generally divided into three stages: constructing an open attribute dictionary, generating dynamic attribute vectors, and aligning and fusing dual-path features.

[0028] (1) First, an open attribute dictionary is constructed, and a large visual language model is used to analyze several example images for each category. Then, carefully designed prompt words are provided to the model. Finally, the model's responses to all categories are collected, deduplicated, and merged to construct an open-vocabulary attribute dictionary.

[0029] (2) Next, dynamic attribute vectors are generated. The open attribute dictionary is converted into cues, and these cues are converted into attribute cue vectors using CLIP's text encoder. A lightweight multimodal large model is introduced, which takes these attribute cues and image regions as input and outputs the confidence of the existence of each attribute through deep visual-semantic reasoning, thus forming dynamic attribute vectors.

[0030] (3) Finally, dual-path feature alignment and fusion are performed. In this stage, the CLIP image encoder is first used to extract visual features and interact with and align the dynamically generated attribute vectors with attention to achieve visual localization of the attributes. Then, the visual features output by CLIP are aligned and fused with the features of the lightweight multimodal large model. Finally, a classifier is used for final prediction.

Claims

1. A visual language zero-shot learning method of open visual attributes, characterized in that The first step is to build an open dictionary; then, a large visual language model is used to analyze several example images for each category; then, specific prompt words are provided to the model; finally, the model's responses to all categories are collected, deduplicated, and merged to form an attribute dictionary of an open vocabulary.

2. The visual language zero-shot learning method of opening visual attributes, characterized by Next, dynamic attribute vectors are generated. The open attribute dictionary is transformed into cues, and these cues are converted into attribute cue vectors using CLIP's text encoder. A lightweight multimodal large model is introduced, which takes these attribute cues and image regions as input. Through deep visual-semantic reasoning, the confidence of the existence of each attribute is output to form dynamic attribute vectors.

3. A visual language zero-shot learning method for opening visual attributes, characterized in that Finally, dual-path feature alignment and fusion are performed. In this stage, visual features are first extracted using the CLIP image encoder and then interacted with and aligned with the dynamically generated attribute vectors to achieve visual localization of the attributes. The visual features output by CLIP are then aligned and fused with the features of a lightweight multimodal large model. Finally, a classifier is used for the final prediction.

4. An open visual attribute visual language zero-shot learning method, which automatically constructs an attribute dictionary and improves the generalization ability of the model, and promotes zero-shot learning to an open world, characterized in that First, an open attribute dictionary is constructed using a visual language model. Then, dynamic attribute vectors are generated using the attribute dictionary. Finally, the CLIP image encoder extracts visual features and interacts with and aligns the dynamically generated attribute vectors to achieve visual localization of attributes. The dual-path features are aligned and fused, and then input into a classifier to obtain the final prediction.

Images