A zero-shot learning method based on aligned variational autoencoders combined with triples
Through the method of combining the alignment variational autoencoder with triples, the potential spatial characteristics are optimized, the problem of unknown category recognition bias in zero-sample learning is solved, and the classification accuracy and model robustness are improved.
Patent Information
- Application Number
- CN202111364293.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-17
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-11-17
AI Technical Summary
When the existing zero-sample learning method lacks training samples for unknown categories, the model is prone to identify unknown categories as known categories, resulting in reduced accuracy and a large gap between generated characteristics and high-quality characteristics.
Using a method based on the combination of alignment variational autoencoder and triplets, the image feature variational autoencoder and attribute variational autoencoder are aligned, and the potential spatial characteristics are optimized through triplet loss to generate high-quality potential space. feature.
It improves the classification accuracy of the model, alleviates the classification bias of unknown categories as known categories, and enhances the robustness and generalization of the model.
Smart Images

Figure CN114022739B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of zero-shot image classification, and in particular relates to a zero-shot learning method based on the combination of an aligned variational autoencoder and a triplet. Background Art
[0002] Deep learning has been widely used in various fields. For example, in image classification problems, its accuracy can currently achieve good results, but its high accuracy is based on supervision. Most deep models are learned through a large number of labeled training samples, focusing on classifying labeled category samples that have appeared in the training process. However, real object categories usually follow a long-tail distribution, in which some categories have abundant training samples, while most other categories have few or even no available training samples. When the model is required to classify these categories with few or no training samples, the original training method is no longer applicable. In addition, the cost of labeling data is very high. Therefore, the research on zero-shot learning is of great significance. There is no intersection between the training set and the test set of zero-shot learning. During training, auxiliary information, such as attributes, is needed to establish the connection between the training set and the test set, so that the model is effective. The training sets of traditional zero-shot learning and generalized zero-shot learning only include known categories. The test set of traditional zero-shot learning only includes unknown categories, while the test set of generalized zero-shot learning includes both known and unknown categories.
[0003] At present, zero-shot learning methods are mainly divided into two categories: spatial embedding-based methods and generative model-based methods. Spatial embedding-based methods can be divided into three categories. The first category is to embed visual features into semantic space, such as the SAE method (Elyor Kodirov, Tao Xiang, and Shaogang Gong. Semantic autoencoder for zero-shot learning. In CVPR, 2017), the second category is to embed semantic features into visual space, such as the UVDS method (Y. Long, L. Liu, L. Shao, F. Shen, G. Ding, and J. Han. From zero-shot learning to conventional supervised classification: Unseen visual data synthesis. In CVPR, 2017), and the third category is to embed visual features and semantic features into a common space, such as the SJE method (Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In CVPR, 2015). The main goal of the spatial embedding-based method is to use a deep network to learn a projection function to map visual features and semantic features to an embedding space. In the generalized zero-shot learning setting, due to the lack of training samples of unknown categories in the training phase, the spatial embedding-based method tends to identify unknown categories as known categories during testing, resulting in reduced accuracy. Therefore, a method based on a generative model was proposed. The generative model can generate visual features for unknown categories for training through semantic features, such as attributes, to make up for the problem of missing training data for unknown categories. At present, the generative model is mainly based on variational autoencoders and generative adversarial networks, such as LisGAN (J.Li, M.Jing, K.Lu, Z.Ding, L.Zhu, and Z.Huang, Leveraging the invariant side of generative zero-shot learning. In CVPR, 2019). However, there is still a gap between the generated features and high-quality features. Summary of the invention
[0004] In order to solve the above problems, the present invention provides a zero-shot learning method based on the combination of aligned variational autoencoders and triplets. This method uses image feature variational autoencoders and attribute variational autoencoders as backbone networks, aligns them, and combines them with triplets to construct a complete model.
[0005] A zero-shot learning method based on an aligned variational autoencoder combined with a triplet, characterized in that the zero-shot learning method uses an image feature variational autoencoder and an attribute variational autoencoder as a backbone network, aligns them, and combines them with the triplet to construct a complete model. The zero-shot learning method specifically includes the following steps:
[0006] S1: Randomly select P known categories from the known category data set, and select K image feature samples for each category. And its corresponding attribute characteristics and label Y, forming a training data set, where N s =P×K, d and d a With dimensions that are image features and attribute features respectively;
[0007] S2, according to a specific triple selection method, is the image feature sample X of S1 s Each sample x in X s Select the qualified positive and negative samples to form a triplet (x, x p ,x n ), and transform the triple (x,x p ,x n ) is input into the model together with the attribute feature a corresponding to x;
[0008] S3, define all encoders and decoders as multi-layer perceptrons with one hidden layer; encoder E of image feature variational autoencoder img and decoder D img The number of neurons in the hidden layer are n_hidden_E_img and n_hidden_D_img respectively; the encoder E of the attribute variational autoencoder att and decoder D att The number of neurons in the hidden layer are n_hidden_E_att and n_hidden_D_att respectively;
[0009] S4,E img The input is the image feature x, and the output is (2×d z )-dimensional vector, the first d z dimension is set to the mean μ of x, and then d z The dimension is set to the variance Σ of x, and d is sampled from a Gaussian distribution (μ, Σ) z dimensional latent space features z~N(0,I); Eatt The input is attribute a, and the output (μ a ,Σ a ), and sample to get z a ~N(0,I),z a The dimension is the same as z; the decoder input is the latent space feature, and the output is the reconstructed input feature;
[0010] S5, calculate the reconstruction loss L1 of the image feature and attribute variational autoencoder according to S4;
[0011] S6, replace E in S4 img and E att The output parameters are aligned and the parameter alignment loss L2 is calculated;
[0012] S7, input the latent space feature z of x in S4 into D att In the process, the latent space feature z of a is a Input to D img , perform cross reconstruction and calculate the cross reconstruction loss L3;
[0013] S8, for the triple (x,x p ,x n ) Calculate the triplet loss based on the reconstruction features of the image feature variational autoencoder, and the triplet loss is calculated to be L4;
[0014] S9, calculate the overall model loss L according to the reconstruction loss L1, parameter alignment loss L2, cross reconstruction loss L3 and triplet loss L4, and perform back propagation optimization on the model parameters;
[0015] S10, using the model learned in S9 to generate latent space features for known and unknown classes;
[0016] S11, train a Softmax classifier in the model latent space, and use the latent space features generated in S10 as the input of the classifier;
[0017] S12, testing the classifier trained in S11 on the test set to obtain the classification accuracy.
[0018] Furthermore, the triple (x,x p ,x n ), x is the anchor point, x p is a positive sample belonging to the same class as the anchor point, (x,x p ) is called a positive sample pair, x n is a negative sample that belongs to a different class from the anchor point, (x,x n ) is called a negative sample pair.
[0019] Furthermore, there are four specific ways to select the specific triples in S2; the first one is to select all possible triplets, and the possible triplets are all triplets that satisfy the condition that the distance between negative sample pairs is less than the distance between positive sample pairs; the second one is to select the most difficult negative sample for each positive sample pair to form a triplet, that is, first fix the anchor point and the positive sample, and select a negative sample from all negative samples that is closest to the anchor point and satisfies the condition that the distance between the anchor point and the positive sample is less than the distance between the positive sample pair; the third one is to randomly select a negative sample for each positive sample pair to form a triplet, that is, first fix the anchor point and the positive sample, and randomly select a negative sample from all negative samples that satisfies the condition that the distance between the anchor point and the positive sample is less than the distance between the anchor point and the positive sample; the fourth one is to select a negative sample of medium difficulty for each positive sample pair to form a triplet, that is, first fix the anchor point and the positive sample, and randomly select a negative sample from all negative samples that satisfies the condition that the distance between the anchor point and the positive sample is less than the distance between the anchor point and the positive sample, but not the negative sample with the smallest distance; wherein the distances between samples are all Euclidean distances.
[0020] Furthermore, in step S5, the reconstruction loss L1 is as follows:
[0021] L1=L re (x,D img (E img (x)))-βKL(Ν(μ,Σ),Ν(0,I))
[0022] +L re (a,D att (E att (a)))-βKL(Ν(μ a ,Σ a ),Ν(0,I))
[0023] Among them, L re is the L1 norm, KL is the KL divergence, N(0,I) is the standard Gaussian distribution; Ν(μ,Σ) and Ν(μ a ,Σ a ) is the output of the image feature encoder and attribute encoder, which obeys Gaussian distribution; D img (E img (x)) is the output of image feature x after being reconstructed by the image feature variational autoencoder; D att (E att (a)) is the output of attribute a after being reconstructed by the attribute variational autoencoder; β is the weight coefficient.
[0024] Furthermore, in step S6, the parameter alignment loss L2 is as follows:
[0025]
[0026] L2=W xa +Wax
[0027] in, is the Frobenius norm.
[0028] Furthermore, in step S7, the cross reconstruction loss L3 is as follows:
[0029] L3=L re (x,D img (E att (a)))+L re (a,D att (E img (x)))
[0030] Among them, D img (E att (a)) and D att (E img (x)) is the output of the model after cross reconstruction.
[0031] Furthermore, in step S8, the triple loss L4 is as follows:
[0032]
[0033] Among them, (x,x p ,x n ) represents the triplet selected according to specific conditions, and d() represents the Euclidean distance.
[0034] Furthermore, in step S9, the overall model loss L is as follows:
[0035] L=L1+αL2+λL3+γL4
[0036] Among them, α, λ, γ are weight coefficients.
[0037] Furthermore, in step S10, for generalized zero-shot learning, the training samples of known categories are divided into two parts, one part Xtrain is used for model learning and classifier training, and the other part Xtest_seen is used for testing the classifier. After the model parameters are fixed, the image feature samples in Xtrain are input into the image feature encoder to generate known category latent space features, and the unknown category attributes are input into the attribute encoder to generate unknown category latent space features.
[0038] Further, in step S12, for generalized zero-shot learning, the test set includes Xtest_seen in claim 9 and samples of unknown categories.
[0039] The beneficial effects of the present invention are:
[0040] (1) The present invention uses a trained attribute variational autoencoder to generate the corresponding latent space features of the unknown category through the unknown category attributes. At the same time, the trained image feature variational autoencoder generates latent space features for the known category image features. Then the classifier is trained with the latent space features and tested on the unknown category samples, which makes up for the problem of missing unknown category training samples in the zero-sample learning training stage, and to a certain extent alleviates the classification bias problem that the model tends to identify unknown categories as known categories during testing, thereby improving the classification accuracy of the model.
[0041] (2) The present invention uses triplet loss for the reconstructed image features, optimizes the distribution of latent space features, and improves the quality of latent space features, thereby making the generated latent space features more conducive to classification.
[0042] (3) The present invention aligns the image feature variational autoencoder and the attribute variational autoencoder through parameter alignment and cross-reconstruction. Parameter alignment can learn the cross-modal joint representation of visual information and semantic information. The use of cross-reconstruction can help the model better capture discriminative information. The joint representation of discriminative visual information and semantic information can improve the robustness and generalization of the model.
[0043] The present invention proposes an aligned variational autoencoder structure to learn and generate cross-modal latent space features, and combines them with triplets to optimize the latent space features, which can significantly improve the robustness and recognition accuracy of the zero-shot learning network model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 Flow chart of the zero-sample learning method in an embodiment of the present invention.
[0045] Figure 2 4 is a network model structure diagram of the zero-sample learning method in an embodiment of the present invention.
[0046] Figure 3 Schematic diagram of sample distribution of reconstructed training samples and real training samples generated by the method in an embodiment of the present invention, wherein (a) is the distribution of real training samples, (b) is the distribution of training samples reconstructed by the model (without the triplet module), and (c) is the distribution of training samples reconstructed by SH-AVAE. DETAILED DESCRIPTION
[0047] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.
[0048] The present invention is a zero-shot learning method based on the combination of an aligned variational autoencoder and a triplet, and the zero-shot learning method specifically comprises the following steps:
[0049] S1: Randomly select P known categories from the known category dataset, and select K image feature samples for each category And its corresponding attribute characteristics and label Y, forming a training data set, where N s =P×K, d and d a The dimensions are image features and attribute features respectively.
[0050] S2: Select image feature samples X for S1 according to a specific triplet selection method s Each sample x in X s Select the qualified positive and negative samples to form a triplet (x, x p ,x n ), triple (x,x p ,x n ) is input into the model together with the attribute feature a corresponding to x. The triple (x,x p ,x n ), x is the anchor point, x p is a positive sample belonging to the same class as the anchor point, (x,x p ) is called a positive sample pair, x n is a negative sample that belongs to a different class from the anchor point, (x,x n ) is called a negative sample pair.
[0051] There are four specific ways to select specific triplets. The first one is to select all possible triplets, that is, all triplets that satisfy the condition that the distance between negative sample pairs is less than the distance between positive sample pairs. The second one is to select the most difficult negative sample for each positive sample pair to form a triplet, that is, first fix the anchor point and the positive sample, and select a negative sample from all negative samples that is closest to the anchor point and satisfies the condition that the distance between the anchor point and the positive sample is less than the distance between the positive sample pair. The third one is to randomly select a negative sample for each positive sample pair to form a triplet, that is, first fix the anchor point and the positive sample, and randomly select a negative sample from all negative samples that satisfies the condition that the distance between the anchor point and the positive sample is less than the distance between the anchor point and the positive sample. The fourth one is to select a negative sample of medium difficulty for each positive sample pair to form a triplet, that is, first fix the anchor point and the positive sample, and randomly select a negative sample from all negative samples that satisfies the condition that the distance between the anchor point and the positive sample is less than the distance between the anchor point and the positive sample, but not the negative sample with the smallest distance. Among them, the distances between samples are all Euclidean distances.
[0052] S3: Because CNN features and properties are already very high-level representations, more hidden layers will reduce performance, so all encoders and decoders are defined as multi-layer perceptrons with one hidden layer. img and decoder D imgThe number of neurons in the hidden layer are n_hidden_E_img and n_hidden_D_img respectively; the encoder E of the attribute variational autoencoder att and decoder D att The number of neurons in the hidden layer are n_hidden_E_att and n_hidden_D_att respectively.
[0053] S4:E img The input is the image feature x, and the output is (2×d z )-dimensional vector, the first d z dimension is set to the mean μ of x, and then d z The dimension is set to the variance Σ of x, and d is sampled from a Gaussian distribution (μ, Σ) z dimensional latent space features z~N(0,I). att Similarly, input attribute a, output (μ a ,Σ a ), and sample to get z a ~N(0,I),z a The dimension is the same as z. The decoder input is the latent space features and the output is the reconstructed input features.
[0054] S5: According to S4, the reconstruction loss L1 of the image feature and attribute variational autoencoder is defined as follows:
[0055] L1=L re (x,D img (E img (x)))-βKL(Ν(μ,Σ),Ν(0,I))
[0056] +L re (a,D att (E att (a)))-βKL(Ν(μ a ,Σ a ),Ν(0,I))
[0057] Among them, L re is the L1 norm, KL is the KL divergence, and N(0, I) is the standard Gaussian distribution. Ν(μ,Σ) and Ν(μ a ,Σ a ) is the output of the image feature encoder and attribute encoder, which obeys Gaussian distribution. img (E img (x)) is the output of image feature x after being reconstructed by the image feature variational autoencoder. D att (E att (a)) is the output of attribute a after being reconstructed by the attribute variational autoencoder. β is the weight coefficient.
[0058] S6: Replace S4 with Eimg and E att The output parameters are aligned and the parameter alignment loss L2 is defined as follows:
[0059]
[0060] L2=W xa +W ax
[0061] in, is the Frobenius norm. ax The calculation formula is similar to that of , only the corresponding parameters are changed.
[0062] S7: Input the latent space feature z of x in S4 into D att In the process, the latent space feature z of a is a Input to D img , perform cross reconstruction, and define the cross reconstruction loss L3 as follows:
[0063] L3=L re (x,D img (E att (a)))+L re (a,D att (E img (x)))
[0064] Among them, D img (E att (a)) and D att (E img (x)) is the output of the model after cross reconstruction.
[0065] S8: For the triples (x,x p ,x n ) Calculate the triplet loss in the reconstruction features of the image feature variational autoencoder, and define the triplet loss as L4 as follows:
[0066]
[0067] Among them, (x,x p ,x n ) represents the triple selected according to specific conditions, and d(x1,x2) represents the Euclidean distance between x1 and x2.
[0068] S9: Calculate the overall model loss L based on the reconstruction loss L1, parameter alignment loss L2, cross reconstruction loss L3 and triplet loss L4, and perform back propagation optimization on the model parameters; the overall loss L is defined as follows:
[0069] L=L1+αL2+λL3+γL4
[0070] Among them, α, λ, γ are weight coefficients.
[0071] S10: Use the model learned in S9 to generate latent space features for known and unknown classes; for generalized zero-shot learning, divide the samples of known categories into two parts, one part Xtrain for model learning and classifier training, and the other part Xtest_seen for classifier testing. After the model parameters are fixed, input the image feature samples in Xtrain into the image feature encoder to generate latent space features for known categories, and input the attributes of unknown categories into the attribute encoder to generate latent space features for unknown categories.
[0072] The models mentioned above refer to the image feature variational autoencoder and the attribute variational autoencoder. Xtrain first trains these two variational autoencoders. After training, only the encoder part of the model is used to generate features, and then use the features to train a classifier, which is equivalent to two stages.
[0073] S11: Train a Softmax classifier in the model latent space, using the latent space features generated in S10 as the input of the classifier.
[0074] S12: Test the classifier trained in S11 on the test set to obtain the classification accuracy. For generalized zero-shot learning, the test set includes Xtest_seen in S10 and samples of unknown categories.
[0075] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.
[0076] A. Experimental conditions
[0077] 1. Experimental Database
[0078] Training and testing are performed on four datasets: AWA1, AWA2, CUB, and SUN. In zero-shot learning, the dataset is divided into training classes (if necessary, further divided into training classes and validation classes) and test classes. Different datasets have different divisions. The training class plus the validation class is the known class, and the test class is the unknown class. There is no intersection between the two. See Table 1 for details.
[0079] Table 1 Detailed introduction of the dataset
[0080]
[0081] 2. Experimental parameter settings
[0082] The fixed parameter settings of the model are shown in Table 2 below:
[0083] Table 2 Model fixed parameters
[0084] n_hidden_E_img n_hidden_D_img n_hidden_E_att n_hidden_D_att P K dz d 1560 1660 1450 660 10 5 64 2048
[0085] B. Experimental results evaluation criteria
[0086] This model is designed for generalized zero-shot learning settings. In generalized zero-shot learning, the test class categories include unknown categories and known categories. Here, the harmonic mean H is used to measure the classification effect. The higher the H value, the better the model effect. The calculation formula of the H value is as follows:
[0087]
[0088] Among them, S is the average classification accuracy of known categories, and U is the average classification accuracy of unknown categories.
[0089] C. Comparative test plan
[0090] This embodiment is compared with other cutting-edge zero-shot learning methods on four data sets.
[0091] Table 3. Comparison of generalized zero-shot image classification performance:
[0092]
[0093] BA-AVAE, BH-AVAE, RH-AVAE and SH-AVAE are all methods proposed in this paper, and the difference lies in the different triple selection conditions. The specific method of selecting triples in BA-AVAE is to select all possible triplets, the specific method of selecting triples in BH-AVAE is to select the most difficult negative sample for each positive sample pair to form a triplet, the specific method of selecting triples in RH-AVAE is to randomly select a negative sample for each positive sample pair to form a triplet, and the specific method of selecting triples in SH-AVAE is to select a medium-difficulty negative sample for each positive sample pair to form a triplet. The results in Table 3 show that the effect of this method exceeds that of other compared methods. ESZSL and SAE are methods based on spatial embedding. Due to the lack of unknown category samples during model training, the model tends to classify unknown categories into known categories during testing. The result shows that S is much higher than U, resulting in a low H value. f-VAEGAN-D2, LisGAN and CADA-VAE are methods based on generative models. Generating training data for unknown categories alleviates classification bias to a certain extent, but there is still a gap between generated features and high-quality features. This method aligns the latent space features of image features and attribute features, more closely links visual features and semantic features, and uses triplet loss to optimize the latent space features, so that the model can generate high-quality latent space features that are more conducive to classification. The data in Table 3 proves the effectiveness of this method from an objective perspective.
[0094] like Figure 3 As shown in the figure, the reconstructed training sample distribution produced by the model with triplet module generated by this method is better than the reconstructed training sample distribution produced by the model without triplet module. This is because the triplet loss optimizes the distribution of latent space features, helps the model learn useful information, and is more conducive to the final classification task.
[0095] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.
Claims
1. A zero-shot learning method based on aligned variational autoencoders combined with triples, Features: The zero-shot learning method uses the image feature variational autoencoder and the attribute variational autoencoder as the backbone network, aligns them, and combines them with the triples to build a complete model. The zero-shot learning method specifically includes the following steps: S1: Randomly select P known categories from the known category data set, and select K image feature samples for each category. And its corresponding attribute characteristics and label Y, forming a training data set, where N s =P×K, d and d a With dimensions that are image features and attribute features respectively; S2, according to a specific triple selection method, is the image feature sample X of S1 s Each sample x in X s Select the qualified positive and negative samples to form a triplet (x, x p ,x n ), and transform the triple (x,x p ,x n ) is input into the model together with the attribute feature a corresponding to x; S3, defines all encoders and decoders as multilayer perceptrons with one hidden layer; Encoder E of the image feature variational autoencoder img and decoder D img The number of neurons in the hidden layer are n_hidden_E_img and n_hidden_D_img respectively; the encoder E of the attribute variational autoencoder att and decoder D att The number of neurons in the hidden layer are n_hidden_E_att and n_hidden_D_att respectively; S4,E img The input is the image feature x, and the output is (2×d z )-dimensional vector, the first d z dimension is set to the mean μ of x, and then d z The dimension is set to the variance Σ of x, and d is sampled from the Gaussian distribution N(μ,Σ) z dimensional latent space features z~N(0,I); E att The input is attribute a, and the output is N(μ a ,Σ a ), and sample to get z a ~N(0,I),z a The dimension is the same as z; the decoder input is the latent space feature, and the output is the reconstructed input feature; S5, calculate the reconstruction loss L1 of the image feature and attribute variational autoencoder according to S4; S6, replace E in S4 img and E att The output parameters are aligned and the parameter alignment loss L2 is calculated; S7, input the latent space feature z of x in S4 into D att In the process, the latent space feature z of a is a Input to D img , perform cross reconstruction and calculate the cross reconstruction loss L3; S8, for the triple (x,x p ,x n ) Calculate the triplet loss based on the reconstruction features of the image feature variational autoencoder, and the triplet loss is calculated to be L4; S9, calculate the overall model loss L according to the reconstruction loss L1, parameter alignment loss L2, cross reconstruction loss L3 and triplet loss L4, and perform back propagation optimization on the model parameters; S10, using the model learned in S9 to generate latent space features for known and unknown classes; S11, train a Softmax classifier in the model latent space, and use the latent space features generated in S10 as the input of the classifier; S12, testing the classifier trained in S11 on the test set to obtain the classification accuracy.
2. According to the zero-shot learning method based on the combination of aligned variational autoencoder and triples according to claim 1, Features: The triple (x,x p ,x n ), x is the anchor point, x p is a positive sample belonging to the same class as the anchor point, (x,x p ) is called a positive sample pair, x n is a negative sample that belongs to a different class from the anchor point, (x,x n ) is called a negative sample pair.
3. According to the zero-shot learning method based on the combination of aligned variational autoencoder and triples according to claim 1, Features: There are four specific ways to select the specific triples in S2; the first one is to select all possible triplets, that is, all triplets that satisfy the condition that the distance between negative sample pairs is less than the distance between positive sample pairs; the second one is to select the most difficult negative sample for each positive sample pair to form a triplet, that is, first fix the anchor point and the positive sample, and select a negative sample from all negative samples that is closest to the anchor point and satisfies the condition that the distance between the negative sample pair and the anchor point is less than the distance between the positive sample pairs; The third type: randomly select a negative sample for each positive sample pair to form a triplet, that is, first fix the anchor point and the positive sample, and randomly select a negative sample from all negative samples that satisfies the distance between the anchor point and the positive sample is less than the distance between the anchor point and the positive sample; The fourth type: select a medium-difficulty negative sample for each positive sample pair to form a triplet, that is, first fix the anchor point and the positive sample, and randomly select a negative sample from all negative samples that satisfies the distance between the anchor point and the positive sample is less than the distance between the anchor point and the positive sample but not the negative sample with the smallest distance; Among them, the distance between samples is the Euclidean distance.
4. According to the zero-shot learning method based on the combination of aligned variational autoencoder and triples according to claim 1, Features: In step S5, the reconstruction loss L1 is as follows: L 1 =L re (x,D img (E img (x)))-βKL(N(μ,Σ),N(0,I)) +L re (a,D att (E att (a)))-βKL(Ν(μ a ,S a ),N(0,I)) Among them, L re is the L1 norm, KL is the KL divergence, N(0,I) is the standard Gaussian distribution; Ν(μ,Σ) and Ν(μ a ,Σ a ) is the output of the image feature encoder and attribute encoder, which obeys Gaussian distribution; D img (E img (x)) is the output of image feature x after being reconstructed by the image feature variational autoencoder; D att (E att (a)) is the output of attribute a after being reconstructed by the attribute variational autoencoder; β is the weight coefficient.
5. According to the zero-shot learning method based on the combination of aligned variational autoencoder and triples according to claim 1, Features: In step S6, the parameter alignment loss L2 is as follows: L 2 =W xa +W ax in, is the Frobenius norm.
6. According to the zero-shot learning method based on the combination of aligned variational autoencoder and triples according to claim 1, Features: In step S7, the cross reconstruction loss L3 is as follows: L 3 =L re (x,D img (AND att (a)))+L re (a,D att (AND img (x))) Among them, D img (E att (a)) and D att (E img (x)) is the output of the model after cross reconstruction.
7. According to the zero-shot learning method based on the combination of aligned variational autoencoder and triples according to claim 1, Features: In step S8, the triplet loss L4 is as follows: Among them, (x,x p ,x n ) represents the triplet selected according to specific conditions, and d() represents the Euclidean distance.
8. According to the zero-shot learning method based on the combination of aligned variational autoencoder and triples according to claim 1, Features: In step S9, the overall model loss L is as follows: L=L 1 +αL 2 +λL 3 +γL 4 Among them, α, λ, γ are weight coefficients.
9. According to the zero-shot learning method based on the combination of aligned variational autoencoder and triples according to claim 1, Features: In step S10, for generalized zero-shot learning, the training samples of known categories are divided into two parts, one part Xtrain is used for model learning and classifier training, and the other part Xtest_seen is used for testing the classifier; after the model parameters are fixed, the image feature samples in Xtrain are input into the image feature encoder to generate known category latent space features, and the unknown category attributes are input into the attribute encoder to generate unknown category latent space features.
10. The zero-shot learning method based on the combination of aligned variational autoencoder and triples according to claim 9, Features: In step S12, for generalized zero-shot learning, the test set includes Xtest_seen and samples of unknown categories.
Citation Information
Patent Citations
Zero sample image classification method based on combination of variational autocoder and adversarial network
CN108875818A
Deep metric learning-based accurate target retrieval method
WO2018137358A1
