Generalized zero sample learning method based on instance-level vision and semantic self-adaption

Abstract

The invention discloses a generalization zero sample learning method based on instance-level vision and semantic self-adaption, which can effectively adapt to pre-trained image features of a GZSL task and realize consistent alignment of attributes and images. This contributes to effective transfer of features between visible classes and invisible classes, prompting the performance representation of zero samples. In terms of semantic adaptation, the method proposes to select attributes specific to instances for dynamic semantic prompt tuning. In terms of visual adaptation, the method constructs an instance visual prototype to generate channel attention, thereby adaptively enhancing key visual features of each sample. A large number of experiments on three GZSL reference data sets show that the method achieves advanced performance.

Description

Technical Field

[0001] This invention relates to the fields of pattern recognition, machine learning, and computer vision, with a particular focus on zero-shot learning techniques. It proposes a zero-shot learning method based on instance-level visual and semantic adaptation. Background Technology

[0002] With the rapid development of artificial intelligence and machine learning technologies, the performance of deep learning models in fields such as image classification and object detection has significantly improved. However, traditional deep learning methods typically rely on large amounts of labeled data, which is difficult to implement in many practical applications. Generalized Zero-Shot Learning (ZSL), as an emerging machine learning paradigm, aims to achieve the recognition of visible and invisible image classes with the assistance of class semantic information. Many methods utilize class attributes or textual descriptions to transfer knowledge from visible classes to invisible classes. Therefore, visual-semantic interaction becomes the core of zero-shot learning, and its main goal is to establish the association between visual features and semantic information. In recent years, various zero-shot learning methods have been proposed and have made significant progress in this field. Based on the different interaction methods, these methods can be broadly divided into two categories: mapping-based methods and generation-based methods.

[0003] With the rapid development of deep learning technology, GZSL has made significant progress in recent years. Early methods used visual encoders to extract global visual features and projected these features and class semantic prototypes into a common space to align the visual and semantic domains. However, these global features may not adequately capture the discriminative details required for fine-grained recognition. To alleviate this problem, subsequent research has utilized attention mechanisms guided by semantic features to focus on local features of the image. These methods typically utilize all available attributes to compute semantically relevant visual attention, which may ignore instance-level variations within each category. For example, certain attributes (e.g., "background pattern entity" and "tail pattern entity") may not be observable in an image due to factors such as different viewpoints, occlusion, and other visual complexities. Therefore, incorporating these unobserved attributes into the visual feature enhancement process may introduce noise. Furthermore, existing transformer-based methods often learn features by focusing on image region features through multi-head self-attention without fully considering the quality of the image region features themselves. If the quality of the area image region features is low, the overall performance of the model may be negatively affected. Summary of the Invention

[0004] To address the aforementioned issues, this invention introduces an Instance-Level Visual and Semantic Adaptation (ILVSA) framework to learn more effective GZSL image representations. The ILVSA framework comprises two main modules: Cue-Based Dynamic Semantic Adaptation (DPSA) and Instance Prototype-Guided Visual Adaptation (IPVA). The DPSA module focuses on adapting image features through dynamic semantic cue adjustments. It dynamically identifies instance-related attributes as semantic cues and then leverages these cues to extract image features through cue-based adjustments, thereby more closely aligning visual and semantic information. On the other hand, the IPVA module is dedicated to refining visual features through an instance prototype-guided attention mechanism. It constructs instance-specific visual prototypes to generate channel attention, which is used to aggregate the most relevant image features, enhancing their discriminative power. By integrating DPSA and IPVA, this framework effectively adapts to pre-trained image features for the GZSL task.

[0005] This invention proposes a novel instance-level visual and semantic adaptation framework for GZSL. Figure 1 An overview of the proposed method is provided, in which a pre-trained Visual Language Model (ViT) serves as the backbone network. ILVSA requires three inputs: CLS, image features, and shared semantic attributes. CLS is randomly initialized at the beginning, image features are obtained from image mapping, and shared semantic attributes are obtained from the natural language model GloVe. This invention is implemented based on three steps: 1) semantic adaptation based on dynamic cues; 2) instance-prototype-guided visual adaptation; and 3) model optimization and inference.

[0006] Step 1: Semantic adaptation based on dynamic prompts;

[0007] Given image features and CLS, these features are directly input into a pre-trained ViT for feature extraction. After multi-head attention is applied to the layers, image features are obtained. Meanwhile, this invention utilizes shared semantic attributes. A semantic cue library was constructed, where shared semantic attributes are encoded using GloVe. Since not all attributes are observable in images, using all attributes as guidance for learning image features may introduce noise. Therefore, we propose dynamically selecting instance-adaptive attributes to guide image feature learning. Specifically, we use CLS features as the query... To select instance-related attributes. This process involves calculating the query image. With hint pool This is achieved through the similarity of attributes:

[0008]

[0009] in, , Then, based on the similarity vector... Select from the suggestion library A tip and By piecing them together, we obtain = After selecting instance adaptive attributes as semantic cues, the learned image features are adjusted using these semantic cues. Similarly, subsequent ViT layers undergo the same semantic adaptation process.

[0010] Step 2: Visual adaptation guided by instance prototypes;

[0011] Image features extracted by pre-trained models are not well-suited for downstream tasks. To learn adaptive visual features, this invention employs instance-level visual adaptation through prototype attention. Specifically, this invention develops instance-level visual prototypes for each image and generates channel attention to select key image features. Specifically, given... Visual markers of layers This invention obtains the instance visual prototype by averaging the features of all visions. :

[0012]

[0013] in Then, the present invention uses two fully connected (FC) layers for channel interaction, and then generates channel attention. This will be the tensor. Copy to extended matrix and use The process of calculating their Hadamard product can be described as follows:

[0014]

[0015]

[0016] in, Indicates regularization, This represents the Sigmoid function. and These are the parameters of the FC layer. Channel attention generated through instance visual prototypes can effectively filter background information and obtain more discriminative visual features. Then, subsequent layers of the ViT model undergo the same visual adaptation process.

[0017] Step 3: Model optimization process;

[0018] After the ViT network completes feature extraction, it obtains the final features. Then, this invention selects CLS for image perception. To this end, this invention proposes a semantic-based classification loss. and semantic alignment loss To optimize ILVSA.

[0019] Semantic-based classification loss:

[0020] Semantic-based classification loss The aim is to effectively classify different categories using category semantic information. Specifically, this invention performs a dot product of CLS features and class semantic prototypes, and then optimizes the parameters using cross-entropy loss. Semantic-based classification loss. Defined as:

[0021]

[0022]

[0023] in express The embedding parameters project class semantics onto the embedding layer in the visual space, forcing the class dimension to be consistent with... The dimensions are consistent.

[0024] Semantic alignment loss:

[0025] To facilitate accurate alignment between image features and their corresponding semantic prototypes, this invention also introduces a semantic alignment loss. ,Should This is achieved by minimizing the distance between them:

[0026]

[0027] Finally, the overall optimization objective can be defined as:

[0028]

[0029] in It controls the semantic alignment loss. The hyperparameters of the weights. Through training, the ILVSA of this invention can align visual and semantic features, thereby achieving the recognition of unseen classes.

[0030] Reasoning stage:

[0031] During the inference process, this method performs visual and semantic adaptation to obtain appropriate image features. Then, the invention uses test images... The CLS marker features are used for classification. This invention applies explicit calibration to balance the predicted probabilities between the visible and invisible classes, as shown in the following formula:

[0032]

[0033] in, It is a hyperparameter used to calibrate the probability bias of predictions for the unknown class, helping to balance the prediction bias between the visible and unknown classes. It is an indicator function, and when The time is zero. Attached Figure Description

[0034] Figure 1 This is a model framework diagram of the ILVSA of the present invention. Detailed Implementation

[0035] To demonstrate the effectiveness of the proposed instance-level visual and semantic adaptive framework, experiments were conducted on three benchmark datasets: Caltech-USCD Birds-200-2011 (CUB), SUN Attribute, and Animalswith Attributes2 (AWA2). The invention splits each dataset into visible and invisible classes according to the Proposed split (PS) setting. In the GZSL setting, the accuracy of test samples from the visible class (denoted as S) and the invisible class (denoted as U) needs to be calculated. To comprehensively evaluate the performance of VSPC in the GZSL setting, the calculation process uses a harmonic mean (H), which balances the accuracy of the visible class S and the accuracy of the invisible class U. The harmonic mean is defined as H = 2 × S × U / (S + U).

Claims

1. A method of instance-level visual and semantic adaptive based generalization zero-shot learning, characterized in that, The implementation steps of the method are as follows: Step one: semantic adaptation based on dynamic prompt; Given the image features and CLS, directly input into the pre-trained ViT for feature extraction; after multi-head attention of the front layer of the ViT, the image features are obtained ； wherein denotes image visual features composed of multiple local visual feature vectors, denotes the CLS visual feature vector of the l-th layer, denotes the n-th local visual feature vector of the l-th layer, and denote the number of visual features and the dimension of visual features, respectively. Using shared semantic attributes Build a semantic prompt library, where shared semantic attributes are encoded by Glove; using all attributes as guidance for image feature learning may bring noise; dynamically select instance adaptive attributes to guide image feature learning; use CLS features as queries to select instance-related attributes; Then according to the similarity vector Select the first Prompt from the prompt library Splice, get = After selecting the instance adaptive attribute as the semantic prompt, the image features are adaptively learned through semantic prompt adjustment; the subsequent ViT layers undergo the same semantic adaptation process; wherein represents the top-K semantic attributes screened out; Step two: visual self-adaptation guided by instance prototype; Instance-level visual adaptation is performed through prototype attention; an instance visual prototype is formulated for each image, and channel attention is generated to select key image features; Given Visual marking of the layer Obtaining an instance visual prototype by averaging all visual features : wherein ; then, channel interaction is performed using two fully connected FC layers, which then produces channel attention ; Through the channel attention generated by the instance visual prototype, the background information is effectively filtered, and more distinctive visual features are obtained; then The subsequent layers of the ViT model undergo the same visual adaptation process; Step three: model optimization process; After the ViT network completes feature extraction, the final feature is obtained ; select CLS for image perception; a semantic-based classification loss and semantic alignment loss are proposed to optimize ILVSA; Semantic-based classification loss The purpose is to effectively classify different categories by using category semantic information; dot product is performed on CLS features and category semantic prototypes, and then parameters are optimized by using cross-entropy loss; Semantic alignment loss: To promote accurate alignment between image features and their corresponding class semantic prototypes, a semantic alignment loss is introduced , the This is achieved by minimizing the distance between them: Finally, the overall optimization objective is defined as: wherein is a hyperparameter controlling the weight of the semantic alignment loss By training, the ILVSA of the present application is able to align visual and semantic features, thus enabling the recognition of unseen classes. Inference stage: visual and semantic self-adaptation is performed during the inference process to obtain appropriate image features; Then, the test image is classified using the CLS label features.

2. The method of claim 1, wherein, In step one, the query image is computed by with the prompt pool similarity of the attributes in the prompt wherein, , ; T is the transpose symbol of a matrix; represents the nth semantic attribute feature in the prompt pool, represents the number of semantic attributes in the prompt pool, represents the similarity vector in the lth layer.

3. The method of claim 1, wherein, In step two, the tensor is copied to the extended matrix and its Hadamard product is computed with expressed as , where, denotes regularization, denotes a Sigmoid function; and are parameters of the FC layer, respectively.

4. The method of claim 1, wherein, In step three, the semantic-based classification loss is defined as: , wherein represents an embedding parameter of the class, an embedding layer that projects the class semantics into the visual space, forcing the dimensionality of the class to be consistent with the dimensionality of .

5. The method of claim 1, wherein, In step three, explicit calibration is applied to balance the prediction probabilities between visible classes and invisible classes, and the formula is as follows: where, is a hyper-parameter to calibrate the probability bias of unknown class prediction, which helps to balance the prediction bias between visible and unknown classes, is an indicator function, and is zero when is one.

Images