An open-domain attribute and object-guided zero-shot learning visual model inference method

Abstract

The present invention proposes an open-domain attribute and object-guided zero-shot learning visual model inference method, comprising the following steps: S1, inputting text information into a text feature encoder #imgabs0# in an AOGN-CZSL model to obtain attribute features and object features based on the text information; inputting image information into an image feature extractor #imgabs1# in an AOGN-CZSL model to obtain attributes and objects based on the image information; S2, feeding the attribute features and object features based on the text information, as well as the attributes and objects based on the image information, into a compatibility learning space P in the AOGN-CZSL model to obtain a trained AOGN-CZSL model; S3, inputting the image or text to be tested into the AOGN-CZSL model to obtain a prediction result. The method can simultaneously process multiple attributes and multiple objects, learning rich dependency structures of different attributes, objects, and their combinations. Furthermore, the encoders learn the connections between attributes and between objects, respectively. The learned dependencies and connections are relatively orderly and simple.

Description

Technical Field

[0001] The present invention relates to the field of visual technology, and in particular to an open domain attribute and object guided zero-shot learning visual model reasoning method. Background Art

[0002] Combinatorial zero-shot learning (CZSL) based on deep learning has become a research hotspot in the field of vision. Combinatorial zero-shot learning is an extended form of zero-shot learning that considers not only individual categories but also combinations of categories. In CZSL, we need to handle not only unseen individual categories but also unseen combinations of categories. For example, if a model has never seen the combination of the attributes "old" and "car" in the training data, but needs to classify "old car" in the test data, this is the task of combinatorial zero-shot learning. CZSL aims to learn how to infer the category of unknown combinations from known attribute and object combinations.

[0003] However, existing combinatorial zero-shot learning methods have several drawbacks: (1) they process each attribute-object combination independently, ignoring the rich dependency structure of different attributes, objects, and their combinations; (2) the learned dependencies and connections are chaotic and complex; and (3) they ignore the relationship between the combination and the image. Summary of the Invention

[0004] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes an open domain attribute and object guided zero-shot learning visual model reasoning method.

[0005] To achieve the above-mentioned object of the present invention, the present invention provides an open domain attribute and object guided zero-shot learning visual model inference method, comprising the following steps:

[0006] S1, input text information into the text feature encoder in the AOGN-CZSL model , obtain attribute features and object features based on text information;

[0007] Input image information into the image feature extractor in the AOGN-CZSL model Get attributes and objects based on image information;

[0008] S2, through the compatibility learning space P in the AOGN-CZSL model, the attribute features and object features based on text information, as well as the attributes and objects based on image information, are fed into the compatibility learning to obtain the trained AOGN-CZSL model;

[0009] S3: Input the image or text to be tested into the AOGN-CZSL model to obtain the prediction result. When the test data is an image, the AOGN-CZSL model outputs the image recognition result; when the test data is text, the AOGN-CZSL model outputs the corresponding image.

[0010] Furthermore, the text information is input into the text feature encoder The following steps are involved:

[0011] S1-1, the attribute tag y in the text information attr and object label y obj Attribute sequence EMB converted into word embedding vector form through word embedding model ATTR , object sequence EMB OBJ ;

[0012] S1-2, the attribute sequence EMB ATTR , object sequence EMB OBJ The data is sent to the encoder, which uses the Encoder module of Transformer.

[0013] Furthermore, the word embedding model is Word2Vec.

[0014] Although word embedding models can use Fasttext, Word2Vec, Glove, etc., Word2Vec can learn word order information, so Word2Vec is selected as the word embedding model.

[0015] Furthermore, the encoding process using the Transformer Encoder module as the encoder is as follows:

[0016] S1-2-1, calculate the degree of attention between attributes and between objects:

[0017] Q=EMBW q (3)

[0018] K=EMBW k (4)

[0019] V=EMBW v (5)

[0020] Here EMB is the embedding representation of the input attribute and object sequence, and EMB takes EMB ATTR or EMB OBJ ;

[0021] W q 、W k 、W v represents the learnable parameter matrix;

[0022] S1-2-2, after obtaining the values ​​of Q (Query), K (Key), and V (Value), the attention coefficient matrix is ​​calculated using formula (6):

[0023]

[0024] Among them A EMB Represents the attention coefficient matrix of the embedding vector EMB;

[0025] K T represents the transposed matrix of K;

[0026] d represents the dimension of EMB;

[0027] is the residual connection;

[0028] S1-2-3, the attention coefficient matrix of the embedding vector EMB is mapped through two layers of linear mapping and activated with a nonlinear activation function:

[0029] EMB hidden =Liner(σ(Liner(A EMB ))) (7)

[0030] Among them EMB hidden Represents the hidden layer features;

[0031] σ is a nonlinear activation function;

[0032] Then EMB hidden With A EMB Perform residual connections and layer normalization:

[0033] EMB′=LayerNorm(EMB hidden +A EMB ) (8)

[0034] Finally, we get the feature matrix of the attribute sequence and object sequence after learning:

[0035] T attr =EMB′ ATTR (9)

[0036] T obj =EMB′ OBJ (10)

[0037] EMB′ ATTR is the embedded representation of the attribute sequence after the Encoder;

[0038] EMB′ OBJ is the embedded representation of the object sequence after the Encoder;

[0039] T attr and T obj They represent the text features of attributes and objects respectively, and they contain attribute-attribute and object-object dependencies.

[0040] Furthermore, the image feature extractor includes a visual converter VIT and a multi-layer perceptron, so the image information is input into the image feature extractor The following steps are involved:

[0041] S1-3, VIT is used as a feature extractor. The input image is passed through VIT to obtain the feature matrix X′:

[0042] X′(X′ l )=FFN(MHA(LayerNorm(X l-1 )+X l-1 ))) (13)

[0043] Where MHA represents the MHA block consisting of multi-head attention mechanism, layer normalization and residual connection;

[0044] FFN represents the FFN block consisting of layer normalization and residual connection;

[0045] LayerNorm(X l-1 ) indicates that the output of the l-1th encoder is normalized;

[0046] X l-1 represents the l-1th encoder;

[0047] S1-3, input X′ into two multi-layer perceptrons respectively to obtain attribute features F attr and object features F obj ;

[0048] F attr =MLP attr (X′) (14)

[0049] F obj =MLP obj (X′) (15)

[0050] Among them F attr It is a multi-layer perceptron MLP with X′ as input attr The attribute characteristics obtained;

[0051] F obj It is a multi-layer perceptron MLP with X′ as input obj Obtained object features.

[0052] Furthermore, the compatibility scoring formula is:

[0053]

[0054] in represents the compatibility scoring function;

[0055] Represents the compatibility score of the triple (I, attr, obj);

[0056] I represents the image sample;

[0057] attr, obj represent attributes and text labels respectively,

[0058] The meaning of bmm is batch matrix multiplication operation;

[0059] S attr and S obj Is the compatibility score between the attribute and the object, and the specific calculation formula is as follows:

[0060]

[0061]

[0062] in represents the compatibility score of the pair (I, attr);

[0063] represents the compatibility score of the pair (I, obj);

[0064] and Represent the extracted attributes and image features F of the object respectively;

[0065] (·) represents the dot product;

[0066] Text features representing learned attributes and objects;

[0067] T, ω, θ, τ, represents the learnable parameter matrix.

[0068] Furthermore, the AOGN-CZSL model adopts the loss function Constraining and Encouraging Image Feature Extractors and text feature encoder Mining visual and textual features of attributes and objects, loss function The expression is as follows:

[0069]

[0070]

[0071]

[0072] Where min means to minimize the loss;

[0073] T is the training set;

[0074] |T| is the number of training elements in the training set;

[0075] exp represents the exponential function with the natural constant e as the base;

[0076] Represents a pair of tuples (I i ,attr i )’s compatibility score;

[0077] Represents a pair of tuples (I i , obj j )’s compatibility score;

[0078] I i is the i-th input image;

[0079] attr i 、obj i Respectively represent I i The true attribute labels and true text labels;

[0080] attr j , obj j Represent all predicted true attribute labels and true text labels respectively;

[0081] attr, obj represent attributes and text labels respectively;

[0082] is the compatibility scoring function;

[0083] ω, θ, τ, represents a learnable parameter;

[0084] y s Indicates visible combinations.

[0085] In summary, due to the adoption of the above technical solution, the present invention has the following advantages:

[0086] (1) It can process multiple attributes and multiple objects simultaneously and learn the rich dependency structures of different attributes, objects and their combinations.

[0087] (2) The encoder learns the connections between attributes and objects respectively. The dependencies and connections learned in this way are more orderly and simple.

[0088] (3) Construct a new multimodal compatibility learning space for images and texts, and embed attributes and objects into the space to perform reasoning learning from visible combinations to invisible combinations.

[0089] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0091] Figure 1 It is a schematic diagram of the overall structure of the AOGN-CZSL model proposed in this invention.

[0092] Figure 2 It is the learning process of the text features of the present invention.

[0093] Figure 3 This is the structure of the image feature extractor of the present invention.

[0094] Figure 4 is the result of the change of encoder parameters on the C-GQA dataset.

[0095] Figure 5 is the result of the change of encoder parameters on the UT-Zappos50k dataset.

[0096] Figure 6 This is the result of the change of encoder parameters on the Clothing16k dataset. DETAILED DESCRIPTION

[0097] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0098] 1. Structure of AOGN-CZSL

[0099] We propose a new solution, namely AOGN-CZSL. Our method is as follows Figure 1 As shown in Figure 2, the two inputs of the AOGN-CZSL model are the sample image (left) and the attribute object text representation (right). Functionally, it can be divided into three components: Image feature extractor Text Feature Encoder And compatibility learning space P. Text feature encoder through encoder with attention mechanism attrand Encoder obj The image feature extractor does not use the traditional convolutional neural network, but is composed of VIT (Vision Transformer) and multi-layer perceptron (MLP). In the compatibility learning space, the compatibility evaluation function is used to evaluate the correlation between image features and text features, and the loss function is optimized. To constrain and encourage image feature extractors and text feature encoder Mining visual and textual features of attributes and objects.

[0100] 1.1 Learning attribute features and object features in text modality

[0101] The Transformer's unique self-attention mechanism models different positions in the input sequence, capturing global dependencies. This self-attention mechanism allows the model to dynamically allocate more attention to relevant positions during encoding, thereby fully understanding the information in the input sequence. By stacking multiple self-attention and feedforward neural network layers, the Transformer encoder effectively captures the global information of the input text and transforms it into a high-dimensional representation for subsequent tasks.

[0102] In our approach, we aim to address the scientific problem of dependencies between different attributes and objects. To this end, we choose to use the Encoder module of Transformer to learn this dependency. The specific learning process is as follows: Figure 2 In this way, we can fully utilize the Transformer’s global understanding of the text during encoding, thereby better solving the dependency problem between different attributes and objects.

[0103] We use Encoder to learn features of the attribute attr and the text label of the object obj. Before feature learning, the attribute label y attr and object label y obj Use pre-trained word embedding (such as word2vec and glove word embedding models) to represent it as emb attr and emb obj , then the attribute sequence and object sequence are:

[0104]

[0105]

[0106] Here N is the total number of attributes and objects in the dataset. The following is the calculation process of the entire text feature encoder module:

[0107] 1.1.1 Calculating the degree of attention

[0108] The calculation of Q(Query), K(Key), and V(Value) is a key step in the attention mechanism. They are used to calculate the degree of attention between the current position and other positions. This degree of attention can be interpreted as the closeness of the connection between different attributes or different objects. For example, the connection between "dog" and "cat" is higher than that between "dog" and "apple" because "dog" and "cat" are both animals. Here, the EMB is expressed by using the embedding of the input attribute and the object sequence. ATTR , EMB OBJ , we can calculate the degree of attention between attributes and between objects:

[0109] Q=EMBW q (3)

[0110] K=EMBW k (4)

[0111] V=EMBW v (5)

[0112] Here EMB is the embedding representation of the input attribute and object sequence, and EMB takes EMB ATTR or EMB OBJ ;W q 、W k 、W v represents the learnable parameter matrix.

[0113] 1.1.2 Generating Attention Coefficient

[0114] After obtaining the values ​​of Q (Query), K (Key), and V (Value), the attention coefficient is calculated using Formula 6:

[0115]

[0116] Among them A EMB Represents the attention coefficient matrix of the embedding vector EMB, K T represents the transposed matrix of K, d represents the dimension of X, is the residual connection.

[0117] 1.1.3 Feedforward Neural Network Layer

[0118] Here FeedForward is actually a two-layer linear mapping activated by a nonlinear activation function:

[0119] EMB hidden =Liner(σ(Liner(AEMB ))) (7)

[0120] Among them EMB hidden represents the hidden layer features, σ is a nonlinear activation function (such as RELU and tanh). We also need to hidden With the A just calculated EMB Perform residual connections and layer normalization:

[0121] EMB′=LayerNorm(EMB hidden +A EMB ) (8)

[0122] We use EMB′∈R batch*seqlen*d Represents the final output result of the Encoder, which is the feature matrix obtained after the attribute sequence or object sequence is learned.

[0123] T attr =EMB′ ATTR (9)

[0124] T obj =EMB′ OBJ (10)

[0125] EMB′ ATTR is the embedded representation of the attribute sequence after the Encoder;

[0126] EMB′ OBJ is the embedded representation of the object sequence after the Encoder;

[0127] T attr and T obj They represent the text features of attributes and objects respectively, and they contain attribute-attribute and object-object dependencies.

[0128] 1.2 Visual feature extraction

[0129] CZSL can be regarded as a multimodal task of Text-Vision. In this section, we will introduce our method for extracting features from images on the Vision side. Figure 3 As shown. We use the VIT (Vision Transformer) model pre-trained on ImageNet and implemented DINO as our image feature extractor Compared with the widely used ResNet18, VIT uses the self-attention mechanism to model the dependencies between different positions in the input sequence, thereby capturing the global information of image I. In contrast, ResNet18 uses fixed-size convolution kernels for local feature extraction and cannot directly obtain the global view.

[0130] 1.2.1 Image Sequence Embedding

[0131] Transformer uses a one-dimensional sequence of token embeddings as input. The original image input to VIT is a two-dimensional image. In order to process 2D images, the original image I needs to be transformed:

[0132]

[0133] X represents the input vector, which is embedded by the sequence I p , object label I c and positional embeds pos Composition. The image I∈R H *W*C Transformed into a patch sequence Where (H, W) is the original resolution of image I (such as 244*244), C is the number of channels of image I (RGB image C=3), (P, P) is the resolution of each patch (such as 16*16), and N' is the total number of patches generated. It is also necessary to convert (P 2 C)→D,→ represents the change, D is a constant latent vector dimension, by I p Just make a linear change:

[0134]

[0135] By putting I p By multiplying it with W, a linear change effect can be achieved.

[0136] 1.2.2 Attributes and Object Visual Features

[0137] Since only VIT is used as a feature extractor in our method, the final output of VIT is the feature matrix X′ extracted from the input image. The final output of VIT can be expressed as:

[0138] X′(X′ l )=FFN(MHA(LayerNorm(X l-1 )+X l-1 ))) (13)

[0139] Among them, MHA represents the MHA block composed of multi-head attention mechanism, layer normalization and residual connection, FFN represents the FFN block composed of layer normalization and residual connection, and both MHA block and FFN block can be repeated l times.

[0140] F attr =MLP attr (X′) (14)

[0141] F attr It is a multi-layer perceptron MLP with X′ as input attr The attribute features obtained are similar to the object features F. obj :

[0142] F obj =MLP obj (X′) ( 1 5)

[0143] like Figure 3 As shown in Figure 1, the feature extraction process is very simple. After the input image I passes through the VIT, the output feature representation X′ is input. X′ then enters the MLP to learn attribute and object features respectively. Although the feature extractor structure is relatively simple, subsequent experiments have shown that it is indeed effective for the CZSL task.

[0144] 1.3 Text-Vision Compatibility Learning Space

[0145] In the previous section, we obtained text features T and visual features F. In this section, we construct a text-vision compatibility learning space P. This is a learnable space that embeds text features T and visual features F into a shared representation space P. In this space, relevant text features and image features are placed close together, while irrelevant text and image features are relatively far apart. By calculating this correlation between text and image features, cross-modal matching and association can be achieved.

[0146] 1.3.1 Compatibility Score

[0147] Here, the correlation is judged by the compatibility score S. The higher the score, the higher the correlation between T and F.

[0148]

[0149] represents the compatibility scoring function, Represents the compatibility score of the triple (I, attr, obj), I represents the image sample, attr, obj represent the attribute and text label respectively, bmm means batch matrix multiplication (BatchMatrix Multiplication) operation, S attr and S obj is the compatibility score between the attribute and the object,

[0150]

[0151]

[0152] in and represents the extracted attributes and image features of the object, (·) represents the dot product, Represents the learned attributes and text features of objects, T,ω,θ,τ, represents the learnable parameter matrix.

[0153] 1.3.2 Optimizing the Loss Function

[0154] For labels (attr, obj) that are closer to sample I, the compatibility score function A higher score will be given to the triple (I, attr, obj), while a triple that deviates more from it will get a lower score. We optimize the attribute loss and object loss The overall loss Jointly learn and train visual feature extractors in an end-to-end manner and text feature encoder

[0155]

[0156]

[0157]

[0158] Where T is the training set, is the compatibility scoring function defined in Equation 16, I i is the input image, (attr i , obj i ) is I i The true label of (attr j , obj j ) represents all possible predictions, ω, θ, τ, represents the learnable parameter, y s Indicates visible combinations.

[0159] 1.3.3 Inference and Prediction

[0160] The purpose of optimizing the loss function is to improve the accuracy of the compatibility scoring function and optimize the visual feature extractor. and text feature encoder , so that more relevant predictions can be assigned higher scores in the inference phase. In the inference phase, for example, we predict a sample image I:

[0161]

[0162] L represents the predicted label (attr, obj) among the triples (I, attr, obj) that obtain the highest compatibility score by searching I through the compatibility score function, where (attr, obj) ∈ Y, which means that our method is also applicable to GCZSL (generalized CZSL).

[0163] The Text-Vision compatibility learning space plays a key role in our approach, enabling text-image association and inference in the CZSL task. This allows accurate zero-shot learning and prediction using the compatibility learning space even when a specific text-image pair has not been seen during training.

[0164] 2. Comparative test

[0165] To evaluate the performance of the proposed AOGN-CZSL, we compare eight deep learning based CZSL methods with AOGN-CZSL:

[0166] (1) SymNet, inspired by group theory, learns symmetries in attribute-object combinations, which can be exploited to gain a deeper understanding of attribute-object pairs;

[0167] (2) CompCos extends CZSL to an open-world environment, considers all possible combinations during reasoning, learns the mapping from image features to the combination semantic space, and calculates the cosine similarity between them. It proposes a feasibility score based on data statistics to remove infeasible combinations.

[0168] (3) GraphEmb proposes using graph convolutional networks to represent attribute-object relationships and combinations, expressing the dependencies between attributes, objects, and combinations in an end-to-end manner;

[0169] (4) Co-CGE improves on GraphEmb, using similar graph convolutional networks and injecting feasibility score additions in the loss layer and graph connections.

[0170] (5) SCEN computes visual prototypes of attributes and objects in a twin contrast space and proposes the use of attribute transformation modules to increase the diversity of training combinations;

[0171] (6) IVR proposes to decompose visual features into concept-invariant domains by masking specific channels of visual features from the perspective of domain generalization.

[0172] (7) OADis is contrary to some previous studies in that it focuses on separating attributes and object features in visual space;

[0173] (8) ADE separates the visual representations of attribute and object features by learning image pairs that share the same attributes or objects, and regards cross attention as a concept disentangler to learn different attribute and object features.

[0174] Experimental results are compared with AOGN-CZSL on three datasets: C-GQA, UT-Zappos50K, and Clothing16k, in both closed-world and open-world scenarios. We primarily use two evaluation metrics for the CZSL task. In the GZSL setting, the model is trained only on seen combinations (labels), resulting in an inherent bias towards unseen combinations. First, to overcome the CZSL model's inherent bias towards unseen combinations, we employ an evaluation protocol that applies different biases to unseen combinations and calculates the Top-1 accuracy for the corresponding seen and unseen combinations. A larger bias means higher unseen accuracy and lower seen accuracy, and vice versa. We calculate the area under the curve (AUC) of the unseen-seen accuracy curve, the best harmonic mean (HM) on the curve, the best seen accuracy (Seen), and the best unseen accuracy (Unseen). AUC is the most representative and stable metric for measuring the performance of the CZSL model. Secondly, in order to observe how the model learns attributes and objects, we give the classification accuracy of attributes and objects (ATTR and OBJ) respectively.

[0175] 2.1 Results on a Closed World

[0176] (1) Results on C-GQA The AOGN-CZSL model outperforms all other state-of-the-art models in the C-GQA dataset under a closed-world environment, as shown in Table 1. Specifically, AOGN-CZSL achieves scores of 18.9 and 5.5 on the most important evaluation metrics, HM (harmonic mean) and AUC (area under the curve), respectively, both exceeding the current best model ADE by 0.9 and 0.3, demonstrating the overall superiority of AOGN-CZSL. Its UNSEEN (prediction ability of unseen samples) performance is comparable to ADE, but in terms of judging sample attributes, AOGN-CZSL achieves an accuracy of 26.7%, far exceeding other models by at least 10 percentage points. This indicates that the AOGN-CZSL model has strong discriminative ability for attribute prediction on C-GQA data. However, in terms of object accuracy, AOGN-CZSL is inferior to all existing models, with a decrease of 17.7% compared to IVR. This is mainly due to the large size and complex number of objects in the C-GQA dataset, and our object encoder cannot well learn the relationship between them. In summary, the overall performance of the AOGN-CZSL model on the C-GQA dataset in a closed-world setting is better than that of existing state-of-the-art methods.

[0177] Table 1. Results of AOGN-CZSL on the C-GQA dataset in a closed-world environment

[0178]

[0179]

[0180] (2) Results on UT-Zappos50K On the UT-Zappos50K dataset, AOGN-CZSL once again demonstrated its outstanding performance on the closed-world CZSL task, as shown in Table 2. First, on the most important evaluation metric, the harmonic mean (HM), the AOGN-CZSL model achieved a high score of 52.5, surpassing all the comparison models and improving by 1.4 over ADE. Second, on the AUC, AOGN-CZSL leads all other models with a result of 38.5. In terms of judging both seen and unseen samples, AOGN-CZSL's accuracy is ahead of other models. For attribute accuracy, AOGN-CZSL does not have a large lead like it does on C-GQA and VAW-CZSL. This is because the attributes in UT-Zappos50k are difficult to distinguish, such as "leather" and "artificial leather". It is difficult for the model to achieve the best performance balance between the validation set and the test set in this dataset. Other baselines also have similar problems. However, AOGN-CZSL performs well overall on UT-Zappos.

[0181] Table 2. Results of AOGN-CZSL on the UT-Zappos50k dataset in a closed-world environment

[0182] UT-Zappos50k AOGN-CZSL ADE OADis IVR SCEN Co-CGE GraphEmb CompCos SymNet HM 52.5 51.1 46.9 49.2 46.7 44.6 47 48.1 45.6 AUC 38.5 35.1 32.6 34.3 30.9 30.8 32.3 31.8 32.6 SEEN 66.4 63.0 60.7 61.5 65.7 60.9 59.1 58.8 60.6 UNSEEN 68.8 64.3 68.8 68.1 62.9 62.6 67.5 63.8 68.6 ATTR 49.0 46.3 49.3 48.4 44.0 46.0 48.1 45.5 48.2 OBJ 75.8 74.0 76.9 74.6 74.4 73.5 75.4 72.4 77.0

[0183] (3) Results on Clothing16k The overall superiority of AOGN-CZSL can also be demonstrated on Clothing16k, as shown in Table 3. Clothing16k is smaller in scale than UT-Zappos50k and is almost noise-free, which is an important reason why we can achieve good results on this dataset. According to the results of the Clothing16k dataset, our model (AOGN-CZSL) performs well in various indicators. On HM, AOGN-CZSL scored 90.7, surpassing all the comparison models. On AUC, AOGN-CZSL scored a high score of 92.6, leading ADE by 0.2 points. In terms of the accuracy of seen samples, AOGN-CZSL reached 98.7%, slightly lower than the 99% of the best model IVR. In terms of attribute accuracy, AOGN-CZSL is on par with the two best models IVR and ADE. Overall, the AOGN-CZSL model continues to demonstrate its strong strength on the Clothing16k dataset, especially in the overall effect evaluation indicators.

[0184] Table 3. Results of AOGN-CZSL on Clothing16k dataset in closed-world environment

[0185] Clothing16k AOGN-CZSL ADE OADis IVR SCEN Co-CGE GraphEmb CompCos SymNet HM 90.7 88.7 86.1 86.6 78.5 87.9 84.2 87.2 79.3 AUC 92.6 92.4 88.4 90.6 78.8 88.3 89.2 90.3 78.8 SEEN 98.7 98.2 97.7 99.0 98.0 98.5 98.0 98.5 98.0 UNSEEN 96.4 97.7 94.2 97.0 89.6 94.7 97.4 96.8 85.1 ATTR 90.2 90.2 84.9 89.3 81.2 87.4 90.0 90.2 75.6 OBJ 89.6 93.6 93.1 93.6 85.4 91.4 93.1 91.8 84.1

[0186] In summary, in complex closed-world scenarios, AOGN-CZSL achieved the best or second-best overall performance on all three datasets, which once again verified its effectiveness in the CZSL task.

[0187] 2.2 Results in the open world

[0188] In open-world scenarios, there are far more attributes and object combinations to consider than in closed-world scenarios. This leads to a significant drop in the performance of existing methods (including ours). We also conducted experimental comparisons of our results with existing state-of-the-art methods in this scenario on the C-GQA, UT-Zappos50K, and Clothing16K datasets.

[0189] (1) Results on C-GQA In the open-world environment, our method achieved a HM of 7.2 and an AUC of 1.4% on the C-GQA dataset, which is superior to other methods. See Table 4 for details. Looking at the accuracy of seen samples, our method achieved 36.1%, surpassing all other methods. Looking at the accuracy of unseen samples, our method achieved 5.1%, which is similar to other methods, indicating that there are still difficulties in generalizing to unseen combinations. In terms of attribute recognition accuracy, our method achieved 33.5%, which is significantly higher than other methods. This shows that our method still has a significant advantage in learning attribute representations even in the complex environment of the open world. For object recognition accuracy, although our method is slightly inferior to other methods, this is because we focus more on modeling attribute relationships and the large number of objects in the C-GQA dataset and its complex connections.

[0190] Table 4. Results of AOGN-CZSL on the C-GQA dataset in an open-world environment. The experimental results of other advanced methods are derived from ADE.

[0191] C-GQA AOGN-CZSL ADE OADis IVR SCEN Co-CGE GraphEmb CompCos SymNet HM 7.2 7.6 4.2 5.7 2.5 3.3 4.8 4.3 4.9 AUC 1.4 1.4 0.7 0.9 0.3 0.5 0.8 0.7 0.8 SEEN 36.1 35.1 33.0 30.6 29.5 31.1 32.7 32.8 30.1 UNSEEN 5.1 4.8 2.6 4.0 1.5 2.1 3.2 2.8 3.2 ATTR 33.5 22.4 14.6 16.9 14.8 15.5 17.2 15.1 18.4 OBJ 24.0 35.6 39.7 36.5 32.3 35.7 36.7 37.8 37.5

[0192] (2) Results on UT-Zappos50K The UT-Zappos50K dataset is simpler than C-GQA and VAW-CZSL. According to our experimental results, see Table 5, we observe that AOGN-CZSL achieves similar performance on this dataset as in the closed-world scenario, achieving a HM of 45.4 and an AUC of 28.8, surpassing all the comparison methods overall and performing the best. We achieved a recognition accuracy of 66.2% for seen samples and a 41.4% for attribute recognition, both of which are higher than those of other methods. Taking all factors into consideration, our method shows the best overall performance on the UT-Zappos50K dataset, with both HM and AUC ranking at the top.

[0193] Table 5. Results of AOGN-CZSL on the UT-Zappos50k dataset in an open-world environment

[0194] UT-Zappos50k AOGN-CZSL ADE OADis IVR SCEN Co-CGE GraphEmb CompCos SymNet HM 45.4 44.8 41.6 42.3 38.0 40.3 40.0 36.0 40.6 AUC 28.8 27.1 25.3 25.3 22.5 22.0 23.5 20.7 25.0 SEEN 66.2 62.4 58.7 60.7 64.8 57.7 60.6 58.1 60.4 UNSEEN 51.3 50.7 53.9 50.0 47.5 43.4 47.0 46.0 51.0 ATTR 41.4 39.9 40.3 38.4 34.9 33.9 37.1 36.4 38.2 OBJ 67.8 71.4 74.7 71.4 73.3 67.2 69.3 71.1 75.0

[0195] (3) Results on Clothing16k According to Table 6, our method also achieves significant advantages on the Clothing16k dataset. Specifically, our overall performance surpasses other methods, especially in the HM indicator, which reaches 76.3. In terms of seen sample recognition, our accuracy is as high as 99.7%, and in terms of unseen sample recognition, we also perform well, obtaining an accuracy of 71.4%. In addition, in attribute recognition, we achieve an accuracy of 78.2%, surpassing all other models. However, our object representation learning is slightly inferior to other methods at an accuracy of 75.5%, which is similar to the situation in the closed world scenario.

[0196] Table 6. Results of AOGN-CZSL on the Clothing16k dataset in an open world environment

[0197] Clothing16k AOGN-CZSL ADE OADis IVR SCEN Co-CGE GraphEmb CompCos SymNet HM 76.3 74.2 63.2 72.0 61.5 69.2 68.3 70.8 68.3 AUC 68.8 68.0 53.4 63.6 53.7 59.3 62.0 64.1 57.4 SEEN 99.7 99.0 98.0 98.7 96.7 98.7 98.5 98.2 98.2 UNSEEN 71.4 73.1 58.6 69.0 62.3 63.8 69.7 69.8 60.7 ATTR 78.2 75.0 57.3 70.3 63.6 68.5 71.8 71.7 57.6 OBJ 75.5 84.5 85.4 84.8 79.1 76.2 82.4 83.7 81.2

[0198] In summary, in the complex open world scenario, AOGN-CZSL achieved the best or second-best overall performance on all three datasets, which once again verified its effectiveness in the CZSL task.

[0199] 2.3 Ablation test

[0200] We designed a variety of ablation experiments to robustly validate our approach and evaluate its effectiveness. All ablation experiments were conducted in a closed-world setting on the UT-Zappos50k, Clothing16k, and C-GQA datasets.

[0201] 2.3.1 Impact of Different Image Feature Extractor Backbone Networks

[0202] We primarily chose Vit as the core image feature extraction framework in our approach, while previous methods commonly use ResNet-18. Furthermore, we experimented with two versions of EfficientNet. Both ResNet and EfficientNet are important convolutional neural network architectures, each with its own strengths in different application scenarios. ResNet is suitable for building deep networks to address the vanishing gradient problem, while EfficientNet excels in resource-constrained environments and is suitable for building small networks with strong performance. Compared to ResNet and EfficientNet, Vit models global dependencies in images and preserves more spatial information, which is why it is effective for our approach. To ensure fairness, we kept other parameters consistent across experiments, such as the learning rate, batch size, and encoder. Table 7 details the results of using five different backbone networks on the UT-Zappos50k, Clothing16k, and C-GQA datasets. While our approach performs similarly when using a CNN-like backbone, using Vit as the backbone achieves the best performance on each dataset. This shows that ViT is indispensable to our approach.

[0203] Table 7. Comparison of the impact of different image feature extractors on our experimental results

[0204]

[0205] 2.3.2 Impact of the Encoder

[0206] In this section, we conduct ablation experiments on the two text feature encoders of AOGN-CZSL and report our experimental results in Table 8. The full version of AOGN-CZSL includes two encoders: attribute encoder attr ) and object encoder (Encoder obj ). We study the impact of different encoder activation conditions under the condition that other parameters are the same.

[0207] Ablation experiments on three datasets using the encoder enabled state show that on the Clothing16k dataset, when the object and attribute encoders are absent, the AOGN-CZSL model achieves a HM and AUC of 66.8 and 65.3, respectively. These metrics improve slightly when the object or attribute encoder is enabled, and rise significantly to 90.7 and 92.6 when both are enabled. On the UT-Zappos50k dataset, when the encoder is absent, the HM and AUC are 48.9 and 35.4, respectively. Enabling either the object or attribute encoder slightly improves these metrics, while enabling both significantly increases them to 53.8 and 39.3. Similar trends are seen on the C-GQA dataset. Without the encoder, the HM and AUC are only 8.4 and 1.5, respectively. However, enabling both encoders significantly improves them to 18.9 and 5.5. Furthermore, enabling the encoder significantly improves both seen and unseen accuracy. In summary, the object and attribute encoders are crucial for improving the key HM and AUC metrics of the AOGN-CZSL model, and also demonstrate significant improvements in judging both seen and unseen examples. This demonstrates the importance and effectiveness of the encoder module in building the AOGN-CZSL model. Overall, the ablation results across three datasets demonstrate the key role of the encoder module in achieving the superior performance of the AOGN-CZSL model.

[0208] Table 8. The effect of the encoder activation on the performance of AOGN-CZSL. "√" represents the encoder is enabled, "×" represents the encoder is not enabled; E attr represents the attribute encoder, E obj Represents an object encoder.

[0209]

[0210] 2.3.3 Impact of Different Word Embeddings

[0211] We selected three popular word embedding models for testing (Fasttext, Word2Vec, and Glove) to observe the impact of different word embeddings on the performance of our model. The specific results are recorded in Table 9.

[0212] Experimenting with different word embeddings on three datasets, we concluded that the Glove word embedding performed best on the Clothing16k dataset, achieving a HM of 90.7 and an AUC of 92.6, significantly outperforming other word embeddings. The Clothing16k dataset has a small vocabulary, but subtle differences in the attributes of different clothing items need to be distinguished, so the Glove word embedding was chosen. This is because Glove learns global word statistics through matrix factorization, enabling it to distinguish subtle differences in vocabulary. This is consistent with Clothing16k's sensitivity to subtle lexical variations. On the UT-Zappos50k and C-GQA datasets, Word2Vec performed best, achieving HMs of 53.8 and 18.9, and AUCs of 39.3 and 5.5, respectively, outperforming other word embeddings. This is consistent with its ability to learn word order information. Using suboptimal word embeddings significantly degraded the metrics across all three datasets, demonstrating the importance of word embedding selection. In addition to HM and AUC, the accuracy of both seen and unseen examples also significantly improved with the optimal word embeddings. In summary, the choice of word embeddings directly affects the performance of AOGN-CZSL on different datasets. Therefore, in terms of optimization effect, it is crucial to choose appropriate word embeddings based on the characteristics of the dataset.

[0213] Table 9. Results of ablation experiments on three popular word vectors

[0214]

[0215] 2.3.4 Impact of Encoder Parameter Selection

[0216] We conducted an experimental analysis on the effectiveness of our method on the encoder. In this section, we experimentally analyzed the impact of changes in two important parameters in the encoder (the number of encoder layers and the number of attention heads). We first conducted an ablation experiment on the number of encoder layers, and tested the experimental results when the number of layers was 1 to 6. At this time, we first fixed the Attention Heads to 6. After finding the optimal number of encoder layers for each dataset, we conducted an ablation experiment on the Attention Heads. At this time, the number of encoder layers was fixed to the optimal value, and then the value of the Attention Heads was changed from 1 to 6 and 10, 12, 15, 20 (the value of the Attention Heads must be guaranteed to be divisible by the input embedding dimension, and our input dimension is 300), and finally found the optimal value on each dataset. We Figures 4 to 6 The experimental results are presented. Figure 4 (a) Figure 5 (a) Figure 5 (a) is the experiment with different encoder layers. Figure 4 (b) Figure 5 (b) Figure 5 (b) shows the experimental results of different attention heads based on the optimal number of encoder layers selected on the left. It includes four indicators: HM, AUC, UNSEE, and SEEN on three datasets: C-GQA, UT-Zappos50k, and Clothing16k.

[0217] (1) Analysis of the impact of encoder parameter selection on the C-GQA dataset We observed that the three datasets have different sensitivities to the changes of these two parameters, among which the C-GQA dataset is the least affected. Figure 4 We can see that changes in both the number of encoder layers and the number of attention heads have little impact on the four metrics, resulting in relatively flat curves. This is because the C-GQA dataset contains a large number of attributes and objects, while our encoder learns relatively stable connections for long sequences of attributes and objects. The optimal parameters for C-GQA were ultimately determined to be a 3-layer encoder and a 2-head multi-attention head.

[0218] (2) Analysis of the influence of encoder parameter selection on the UT-Zappos50k dataset For the UT-Zappos50k dataset, we start from Figure 5 The fluctuations in the line graph show that it is far more sensitive to these two parameters than C-GQA, with the most significant changes in HM, AUC, and UNSEEN. Experiments have shown that 6 encoder layers and 4 attention heads are the optimal values ​​for these two parameters for AOGN-CZSL on UT-Zappos50k. We found that HM and AUC reached their optimal values ​​at 20 attention heads, but we did not use this option. This is because the improvement in results with 4 attention heads is not significant, while the increase in parameters is significant.

[0219] (3) Analysis of the impact of encoder parameter selection on the Clothing16k dataset Clothing16k is more obviously affected by these two parameters, e.g. Figure 6 (a) The changes in HM, AUC, and UNSEEN are most pronounced when the number of encoder layers changes from 2 to 4. Because the Clothing16k dataset is a small dataset with a small number of attributes and objects, changing the encoder parameters leads to unstable results. The optimal results on Clothing16k are achieved with 6 encoder layers and 3 attention heads.

[0220] We also found a pattern from these three experiments: the three indicators HM, AUC and UNSEEN, except SEEN, have almost the same fluctuation. This situation is more obvious in the experiments with the number of encoder layers, for example Figure 5 (a) and Figure 6 (a). In our method, complex datasets are less affected by the number of encoder layers and the number of attention heads, while smaller and simpler datasets are more affected.

[0221] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. An open domain attribute and object guided zero-shot learning visual model inference method, characterized by: The following steps are involved: S1, feeding text information into the text feature encoder in the open-domain attribute and object-guided zero-shot learning AOGN-CZSL model Obtain attribute features and object features based on text information; Input image information into the image feature extractor in the AOGN-CZSL model Get attributes and objects based on image information; The step of inputting text information into a text feature encoder 𝒯 comprises the following steps: S1-1, the attribute tag y in the text information attr and object label y obj Attribute sequence EMB converted into word embedding vector form through word embedding model ATTR , object sequence EMB OBJ ; S1-2, the attribute sequence EMB ATTR , object sequence EMB OBJ The data is sent to the encoder, which uses the Encoder module of Transformer; The encoding process using the Transformer Encoder module as the encoder is as follows: S1-2-1, calculate the degree of attention between attributes and between objects: Q=EMBW q (3) K=EMBW k (4) V=EMBW v (5) Here EMB is the embedding representation of the input attribute and object sequence, and EMB takes EMB ATTR or EMB OBJ ; W q 、W k 、W v represents the learnable parameter matrix; S1-2-2, after obtaining the values ​​of Q (Query), K (Key), and V (Value), the attention coefficient matrix is ​​calculated using formula (6): Among them A EMB Represents the attention coefficient matrix of the embedding vector EMB; K T represents the transposed matrix of K; d represents the dimension of EMB; is the residual connection; S1-2-3, the attention coefficient matrix of the embedding vector EMB is mapped through two layers of linear mapping and activated with a nonlinear activation function: EMB hidden =Liner(σ(Liner(A EMB ))) (7) Among them EMB hidden Represents the hidden layer features; σ is a nonlinear activation function; Then EMB hidden With A EMB Perform residual connections and layer normalization: EMB′=LayerNorm(EMB hidden +A EMB ) (8) Finally, we get the feature matrix of the attribute sequence and object sequence after learning: T attr =EMB′ ATTR (9) T obj =EMB′ OBJ (10) EMB′ ATTR is the embedded representation of the attribute sequence after the Encoder; EMB′ OBJ is the embedded representation of the object sequence after the Encoder; T attr and T obj The text features representing attributes and objects respectively, which contain attribute-attribute and object-object dependencies; The AOGN-CZSL model adopts the loss function Constraining and Encouraging Image Feature Extractors and text feature encoder Mining visual and textual features of attributes and objects, loss function The expression is as follows: Where min means to minimize the loss; |T| is the number of training elements in the training set; exp represents the exponential function with the natural constant e as the base; Represents a pair of tuples (I i ,attr i )’s compatibility score; Represents a pair of tuples (I i ,obj j )’s compatibility score; I i is the i-th input image; attr i 、obj i Respectively represent I i The true attribute labels and true text labels; attr j ,obj j Represent all predicted true attribute labels and true text labels respectively; attr, obj represent attributes and text labels respectively; is the compatibility scoring function; ω, θ, τ, represents a learnable parameter; y s Indicates visible combinations; S2, through the compatibility learning space P in the AOGN-CZSL model, the attribute features and object features based on text information, as well as the attributes and objects based on image information, are fed into the compatibility learning to obtain the trained AOGN-CZSL model; S3: Input the image or text to be tested into the AOGN-CZSL model to obtain the prediction result.

2. The open domain attribute and object guided zero-shot learning visual model inference method according to claim 1, characterized in that: The word embedding model is Word2Vec.

3. The open domain attribute and object guided zero-shot learning visual model inference method according to claim 1, characterized in that: The image feature extractor includes a visual converter VIT and a multi-layer perceptron. Therefore, the image information is input into the image feature extractor. The following steps are involved: S1-3, VIT is used as a feature extractor. The input image is passed through VIT to obtain the feature matrix X′: X′(X′ l )=FFN(MHA(LayerNorm(X l-1 )+X l-1 ))) (13) Where MHA represents the MHA block consisting of multi-head attention mechanism, layer normalization and residual connection; FFN represents the FFN block consisting of layer normalization and residual connection; LayerNorm(X l-1 ) indicates that the output of the l-1th encoder is normalized; X l-1 represents the l-1th encoder; S1-3, input X′ into two multi-layer perceptrons respectively to obtain attribute features F attr and object features F obj ; F attr =MLP attr (X′) (14) F obj =MLP obj (X′) (15) Among them F attr It is a multi-layer perceptron MLP with X′ as input attr The attribute characteristics obtained; F obj It is a multi-layer perceptron MLP with X′ as input obj Obtained object features.

4. The open domain attribute and object guided zero-shot learning visual model inference method according to claim 1, characterized in that: The compatibility scoring formula is: in represents the compatibility scoring function; Represents the compatibility score of the triple (I, attr, obj); I represents the image sample; attr, obj represent attributes and text labels respectively; The meaning of bmm is batch matrix multiplication operation; S attr and S obj Is the compatibility score between the attribute and the object, and the specific calculation formula is as follows: in represents the compatibility score of the pair (I, attr); represents the compatibility score of the pair (I, obj); and Represent the extracted attributes and image features F of the object respectively; (·) represents the dot product; Text features representing learned attributes and objects; ω, θ, τ, represents the learnable parameter matrix.

Images