A Generative Zero-Shot Learning Method Based on Visual Semantic Constraints

Abstract

The present invention relates to the field of image classification in computer vision, and discloses a method for generating zero-shot learning based on visual semantic constraints. First, it extracts the visual features of images of known classes, the semantic features of attributes, and the semantic features of attributes of unknown classes, establishes the visual semantic constraints between the generated features generated by the generator based on the semantic features of known classes and the known classes and unknown classes, constructs a generative adversarial network based on visual semantic constraints, and trains to obtain a generator. Then, using the trained generator, it obtains the generated features of unknown classes. Finally, it integrates the generated features of unknown classes into the known class feature set and trains the zero-shot classifier, which improves the recognition accuracy of the model for unknown samples by improving the quality of the generated features of unknown classes. It is applicable to zero-shot learning in fields such as endangered animal recognition, security inspection prediction, face verification, and fault diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of image classification in computer vision, and particularly to a method for generating zero-shot learning based on visual semantic constraints. Background Art

[0002] Zero-shot learning is to use the visual features of known classes and auxiliary information (class descriptions) to train a classification model, which can correctly predict unknown class samples that have not participated in training during the testing process. Zero-shot learning plays an important role in endangered animal identification, security inspection prediction, face verification, and fault diagnosis fields.

[0003] Currently, zero-shot learning is mainly divided into attribute prediction, feature mapping, and feature generation methods. Among them, the method based on attribute prediction mainly calculates the attribute probability of an image to infer the label of the unknown class with the highest probability. The attributes annotated for each class are used as auxiliary information for the class. IAP first learns an attribute classifier to calculate the probability of the input image to each attribute, and then calculates the probability from each attribute to the class label and takes the label with the highest probability as the predicted label of the image.

[0004] The method based on feature mapping mainly maps visual features and semantic features to the same space, and then judges the similarity between semantic features and visual features in this space. DeVISE learns the mapping function from image features to the semantic space and uses the SVM pairwise ranking loss for similarity metric learning. ConSE first classifies the input image, stacks the semantic features of the top k known classes with the highest probability together with the image mapping features as the final mapping vector, and then performs feature metric in the semantic space.

[0005] Compared with the above methods, the method based on feature generation can generate visual features of unknown classes and reduce the problem of model bias towards known class samples caused by sample imbalance. The method based on feature generation mainly uses known classes to train a generation model (GAN, VAE, etc.), and then generates visual features of unknown classes to learn a zero-shot classifier, so as to predict unknown classes. f-CLSWGAN uses the semantic features of unknown classes to generate visual features of unknown classes, and learns a zero-shot classifier through the generated visual features to predict unknown classes. CVAE constructs two autoencoders to perform feature encoding and decoding from visual and semantic modalities respectively, and combines the common information of visual and feature two types of features to improve the representation ability of the generated features.

[0006] Existing zero-shot learning methods can achieve good prediction results, but using known class data to train the model to test unknown classes is prone to overfitting, and it is difficult to generate high-quality unknown class features, resulting in poor prediction performance of the zero-shot classifier. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to propose a generation zero - shot learning method based on visual semantic constraints, improve the quality of generated features for unknown classes, and thus improve the recognition accuracy of the model for unknown samples.

[0008] The technical solution adopted by the present invention to solve the above - mentioned technical problem is as follows:

[0009] A generation zero - shot learning method based on visual semantic constraints, comprising the following steps:

[0010] A. Train a zero - shot classifier model;

[0011] A1. Data preparation

[0012] Construct a training data set, the samples of the training data set include known - class samples and unknown - class samples, the known - class samples include images and their class attributes and labels, and the unknown - class samples include class attributes and labels;

[0013] Based on the known - class samples, extract the visual feature x i from the image, and extract the semantic feature y i according to the class attribute; based on the unknown - class samples, extract the semantic feature

[0014] A2. Construct a generative adversarial network, train the generative adversarial network based on the known - class samples to obtain a trained generator; moreover, construct visual - semantic constraints between the generated features generated by the generator and the known - class samples and the unknown - class samples, and constrain the training process of the generator;

[0015] A3. Input the semantic feature of the unknown - class samples into the trained generator G to generate the visual feature of the unknown class Then, combine the generated visual feature of the unknown class with the extracted visual feature x i of the known class to obtain the complete visual feature set V of the training data set; or, input the semantic feature y i of the known - class samples and the semantic feature of the unknown - class samples into the trained generator G to generate the visual feature v n and obtain the complete visual feature set V of the training data set;

[0016] A4. Use the visual features and labels of the samples in the training data set to train the classifier to obtain a trained classifier;

[0017] B. Based on the trained classifier, perform a detection task:

[0018] Extract the visual features of the image to be detected, input them into the trained classifier for image classification, and output the class label of the image.

[0019] Furthermore, in step A2, the training of the generative adversarial network includes:

[0020] A21. The generator G obtains the generated features i according to the semantic feature y of the known class. Fix the parameters of the generator G, and obtain the maximum training loss of the discriminator D according to the generated features and perform backpropagation training on the discriminator D;

[0021] A22. The generator G obtains the generated features i according to the semantic feature y of the known class. Fix the parameters of the discriminator D, and obtain the minimum training loss of the generator G according to the generated features and perform backpropagation training on the generator G under the constraint of visual semantic constraint;

[0022] A23. Repeat steps A21 - A22 until the generative adversarial network converges or reaches the preset number of training rounds to obtain the trained generator.

[0023] Specifically, in step A21, the loss function of the discriminator D is:

[0024]

[0025] Among them, represents the maximum training loss of the discriminator D, E represents the mathematical expectation, λ is the weight coefficient, x i is the visual feature x i extracted from the i-th known class image, y i is the semantic feature extracted from the class attribute of the i-th known class image, is the generated feature obtained by the generator according to the semantic feature y i , is the second norm expected by the discriminator D, is the visual feature x i and the generated feature interpolation between.

[0026] Specifically, in step A22, the loss function of the generator G is:

[0027]

[0028] Among them, It represents minimizing the training loss of the generator G. E represents the mathematical expectation, and y i is the semantic feature extracted according to the category attribute of the i-th known-class image, and the generated feature obtained by the generator according to the semantic feature y i is the generated feature, L vs is the visual-semantic constraint between the generated feature of the generator and the known-class samples as well as the unknown-class samples. γ is a hyperparameter.

[0029] Furthermore, in step A2, the visual-semantic constraint between the generated feature generated by the generator and the known-class samples as well as the unknown-class samples is:

[0030] L vs = L s + L u

[0031] The above L s is the visual-semantic constraint between the generated feature and the known-class samples, and the calculation formula is as follows:

[0032]

[0033] where δ is a hyperparameter, is the generated feature obtained by the generator according to the semantic feature y of the i-th known-class category attribute i is the generated feature, is the visual feature center of the i-th known-class category, is randomly selected from the visual feature centers of other known-class categories except for that, and ||·||2 represents the Euclidean norm;

[0034] The above L u is the visual-semantic constraint between the generated feature and the unknown-class samples, and is obtained according to the following formula

[0035]

[0036] where δ is a hyperparameter, is the generated feature obtained by the generator according to the semantic feature y of the i-th known-class category attribute i is the generated feature, is the visual feature center of the i-th known-class category, is the generated feature obtained by the generator according to the semantic feature of a randomly selected unknown-class category attribute is the generated feature, and ||·||2 represents the Euclidean norm.

[0037] Furthermore, the visual feature center of the category is calculated according to the following formula

[0038]

[0039] Among them, p is the number of samples in the i-th category of the known classes, and x j is the visual feature of the j-th image in this category.

[0040] Furthermore, in step A3, traverse the semantic features of each unknown class sample and input them into the trained generator G respectively to generate the visual features of the unknown class Then, the generated visual features of the unknown class are merged with the extracted visual features x of the known class i to obtain the complete visual feature set V of the training dataset.

[0041] Specifically, in step A4, the classifier is a Softmax classifier, and the classifier is trained. Its loss function is:

[0042]

[0043] Among them, P n represents the true class label corresponding to the n-th visual feature of the visual feature set, and Q(c l |v n ) represents the probability that the classifier predicts it as the class label c n according to the n-th visual feature v of the visual feature set l , and m is the number of samples in batch training.

[0044] The beneficial effects of the present invention are:

[0045] Based on the generative adversarial network (GAN), the present invention designs a visual semantic constraint loss. By calculating the inter-class differences between the generated visual features, the true visual features of the known classes, and the generated features of the unknown classes, the inter-class distribution of the features is optimized, the inter-class distinguishability of the generated features is improved, thereby significantly improving the recognition accuracy of the unknown class samples, and further improving the prediction effect of the zero-shot classifier. Description of the Drawings

[0046] Figure 1 It is a model framework diagram of the zero-shot learning method based on visual semantic constraint of the present invention. Detailed Embodiments

[0047] The present invention aims to propose a zero-shot learning method based on visual semantic constraint to improve the quality of the generated features of the unknown classes, thereby improving the recognition accuracy of the model for the unknown samples.

[0048] The core idea is as follows: First, train a generator to generate visual features of unknown-class samples for the classifier to recognize. Then, combine the generated features of the unknown-class samples and the visual features of the known-class samples as the training sample set for the classifier. Moreover, during the training process of the generator, construct visual semantic constraints between the generated features and the visual features of the known-class and unknown-class samples respectively to train the generation model and enhance the quality of the generated features of the unknown class. Therefore, the zero-shot classifier obtained based on the present invention can improve its recognition accuracy for unknown-class samples.

[0049] Embodiment:

[0050] Such as Figure 1 As shown, the method for generating zero-shot learning based on visual semantic constraints in this embodiment mainly includes data preprocessing, training a generator based on visual semantic constraints, generating visual features of unknown classes and training a zero-shot classifier, and performing a detection task based on the zero-shot classifier model. The specific implementation processes of each step are elaborated in detail below:

[0051] Step 1. Data preprocessing:

[0052] This step mainly involves the construction of a training set and feature extraction. First, construct a training data set. The samples of the training data set include known-class samples and unknown-class samples. The known-class samples include images and their class attributes and labels. The unknown-class samples include class attributes and labels. Then, based on the known-class samples, extract visual feature x i from the image, and extract semantic feature y i from the class attribute; based on the unknown-class samples, extract semantic feature

[0053] In the learning of the zero-shot classifier of the present invention, the known class directly uses its visual features for model training, while the unknown class uses the visual features generated according to the semantic features of its class attributes for model training. Therefore, in the present invention, the definition of the unknown class is: the class corresponding to the image and its visual features not participating in the classifier training is called the unknown class.

[0054] Furthermore, in this embodiment, it specifically includes:

[0055] Step 1.1. Use the CUB public data set as the training data set, which has 11,788 bird images, a total of 200 classes, including 150 known classes and 50 unknown classes.

[0056] Step 1.2. Based on the known-class samples, extract visual feature x i from the image, construct a visual feature set X, and extract semantic feature y i, form the semantic feature set Y of known classes; based on unknown class samples, extract semantic features according to class attributes Form the semantic feature set of unknown classes

[0057] Among them, for the extraction of image visual features, the pre-trained ResNet-101 network on the ImageNet dataset is used, and the 2048-dimensional features output by the pooling layer are used as the visual features of the image; for the CUB public dataset, each class has 312 attributes, and the semantic features of class attributes are obtained by annotation encoding based on continuous attributes.

[0058] Step 2: Train the generator based on visual-semantic constraints

[0059] In this step, by constructing a generative adversarial network, the generative adversarial network is trained based on known class samples to obtain a trained generator; moreover, the visual-semantic constraints between the generated features generated by the constructed generator and known class samples as well as unknown class samples are used to constrain the training process of the generator.

[0060] Furthermore, the generator G and discriminator D of the generative adversarial network are trained in an alternating adversarial manner, and by establishing the visual-semantic constraints between the generated features of the generator and the visual features of known classes and unknown classes, a generative adversarial network based on visual-semantic constraints is constructed to enhance the difference between the generated features and non-homogeneous visual features and improve the distinguishability between the generated features, specifically including:

[0061] Step 2.1: Construct visual-semantic constraints

[0062] Specifically, the visual-semantic constraints between the generated features generated by the generator and known class samples as well as unknown class samples are:

[0063] L vs = L s + L u

[0064] The above L s is the visual-semantic constraint between the generated features and known class samples, and the calculation formula is as follows:

[0065]

[0066] Among them, δ is a hyperparameter, is the generated feature obtained by the generator according to the semantic feature y of the i-th known class category attribute i , is the visual feature center of the i-th known class, is from A visual feature center is randomly selected from the visual feature centers of other known classes except , and ||·||2 represents the two-norm;

[0067] The above L u To generate visual semantic constraints between features and unknown class samples, the following formula is used to obtain

[0068]

[0069] Among them, δ is a hyperparameter, is the semantic feature y generated by the generator based on the attribute of the i-th known class category i The generated features obtained, is the visual feature center of the i-th known category, is the semantic feature of an unknown class attribute randomly selected by the generator The generated features obtained, ||·||2 represents the two-norm.

[0070] And calculate the visual feature center of the category according to the following formula

[0071]

[0072] Among them, p is the number of samples of the i-th category in the known class, x j is the visual feature of the jth image in this category.

[0073] Step 2.2: Train the generator based on visual semantic constraints, specifically including:

[0074] 2.21. The generator G is based on the semantic features y of the known class i , and generate features Fixed the parameters of the generator G, according to the generated features Obtain the maximum training loss of the discriminator D and perform backpropagation training on the discriminator D.

[0075] The loss function of the discriminator D is:

[0076]

[0077] in, Represents the maximum training loss of the discriminator D, E represents the mathematical expectation, λ is the weight coefficient, x i is the visual feature x extracted from the i-th known class image i ,y i is the semantic feature extracted based on the category attribute of the i-th known class image, The generator is based on the semantic feature y i The generated features obtained, is the expected second norm of the discriminator D, is the visual feature x i and the generated feature is the interpolation between them.

[0078] 2.22. The generator G generates the generated feature i according to the semantic feature y of the known class. Fix the parameters of the discriminator D, and obtain the training loss that minimizes the generator G according to the generated feature . Under the constraint of the visual-semantic constraint, perform backpropagation training on the generator G.

[0079] The loss function of the generator G is:

[0080]

[0081] where represents minimizing the training loss of the generator G, E represents the mathematical expectation, y i is the semantic feature extracted according to the class attribute of the i-th known-class image, is the generated feature obtained by the generator according to the semantic feature y i , L vs is the visual-semantic constraint between the generated feature of the generator and the known-class samples and the unknown-class samples, and γ is a hyperparameter.

[0082] 2.23. Repeat steps 2.21 - 2.22 until the generative adversarial network converges or reaches the preset number of training epochs, and obtain the trained generator.

[0083] Step 3: Generate visual features of unknown classes and train the zero-shot classifier

[0084] In this step, first, based on the semantic feature of the unknown-class samples, input it into the trained generator G to generate the visual feature of the unknown class. Then, merge the generated visual feature of the unknown class with the extracted visual feature x i of the known class to obtain the complete visual feature set V of the training dataset; or, based on the semantic feature y i of the known-class samples and the semantic feature of the unknown-class samples, input them into the trained generator G to generate the visual feature v n to obtain the complete visual feature set V of the training dataset.

[0085] In this embodiment, traverse the semantic features of each unknown-class sample and input them into the trained generator G respectively to generate the visual feature Then, the visually characterized features of the generated unknown classes are merged with the visually characterized features x of the extracted known classes i to obtain the complete visually characterized feature set V of the training data set. And according to the training requirements of the classifier, the number of visually characterized features of the unknown classes generated by the generator is set .

[0086] Then, using the visually characterized features and their labels of the training data set samples, the classifier is trained to obtain a completely trained classifier.

[0087] For the class labels, in the CUB public data set, the known classes have class labels; while for the unknown classes in the present invention, the samples of the known classes in the CUB public data set can be used to mask their images, thereby constituting the unknown classes during the training of the present invention; secondly, in the CUB public data set, the class attributes of the unknown classes are generated according to the attribute combinations of the known classes. Therefore, their class labels can also be generated while generating their class attributes; of course, the class labels of the unknown classes can also be obtained by manual indexing. Since there are only 50 classes of unknown classes in the CUB public data set, in this embodiment, the class labels of the unknown classes are obtained by manual indexing.

[0088] In this embodiment, the classifier is a Softmax classifier. When training the classifier, its loss function is:

[0089]

[0090] where P n represents the true class label corresponding to the nth visually characterized feature of the visually characterized feature set, and Q(c l |v n ) represents the probability that the classifier predicts that the class label is c n based on the nth visually characterized feature v l of the visually characterized feature set, and m is the number of samples in the batch training.

[0091] Step 4: Perform detection tasks based on the zero-shot classifier model

[0092] In this step, by extracting the visually characterized features of the image to be detected and inputting them into the completely trained classifier for image classification, the class label of the image is output.

[0093] Although the present invention has been described herein with reference to embodiments of the present invention, the above embodiments are only preferred embodiments of the present invention, and the embodiments of the present invention are not limited by the above embodiments. It should be understood that those skilled in the art can design many other modifications and embodiments, which will fall within the scope of the principles and spirit disclosed in this application.

Claims

1. A zero-shot learning generation method based on visual semantic constraints, characterized in that It includes the following steps: A. Train a zero-shot classifier model; A1. Data preparation Construct a training data set. The samples in the training data set include known-class samples and unknown-class samples. The known-class samples include images and their class attributes and labels, and the unknown-class samples include class attributes and labels; Based on known class samples, extract visual feature x according to the image i , extract semantic feature y according to the class attribute i ; Based on unknown class samples, extract semantic features according to the class attribute A2. Construct a generative adversarial network, train the generative adversarial network based on the known-class samples to obtain a trained generator; and construct visual semantic constraints between the generated features generated by the generator and the known-class samples and between the unknown-class samples to constrain the training process of the generator; In step A2, the visual semantic constraints between the generated features generated by the generator and the known-class samples and between the unknown-class samples are: L vs = L s + L u The above-mentioned L s To generate the visual semantic constraint between the features and the known class samples, the calculation formula is as follows: where δ is a hyperparameter, is the generated feature obtained by the generator according to the semantic feature y of the i-th known class category attribute i and is the visual feature center of the i-th known class category, is a visual feature center randomly selected from the visual feature centers of other known class categories other than and ‖·‖2 represents the Euclidean norm; The above-mentioned L u is to generate the visual semantic constraint between the features and the unknown class samples, and obtain according to the following formula where δ is a hyperparameter, is the generated feature obtained by the generator according to the semantic feature y of the i-th known class category attribute i and is the visual feature center of the i-th known class category, is the generated feature obtained by the generator according to the semantic feature of a randomly selected unknown class category attribute and ‖·‖2 represents the Euclidean norm. A3. Semantic features based on unknown class samples Input into the trained generator G to generate visual features of the unknown class Then, the generated visual features of the unknown class Are merged with the extracted visual features x of the known class i To obtain the complete visual feature set V of the training dataset; or, based on the semantic feature y of the known class samples i And the semantic features of the unknown class samples Input into the trained generator G to generate visual features v n To obtain the complete visual feature set V of the training dataset; A4. Use the visual features and their labels of the training data set samples to train the classifier to obtain a trained classifier; B. Based on the trained classifier, perform a detection task: Extract the visual features of the image to be detected, input them into the trained classifier for image classification, and output the class label of the image.

2. The zero-shot learning generation method based on visual semantic constraints according to claim 1, characterized in that In step A2, the training of the generative adversarial network includes: A21. The generator G obtains a generated feature according to the semantic feature y of the known class i , and gets the generated feature Fix the parameters of the generator G, and according to the generated feature obtain the training loss that maximizes the discriminator D, and perform backpropagation training on the discriminator D; A22. The generator G obtains the generated feature according to the semantic feature y of the known class i , and obtains the generated feature Fix the parameters of the discriminator D. According to the generated feature Obtain the training loss that minimizes the generator G, and under the constraint of the visual semantic constraint, perform backpropagation training on the generator G; A23. Repeat steps A21 - A22 until the generative adversarial network converges or reaches a preset number of training epochs to obtain a trained generator.

3. The zero-shot learning method based on visual semantic constraints according to claim 2, wherein In step A21, the loss function of the discriminator D is: Among them, represents maximizing the training loss of the discriminator D, E represents the mathematical expectation, λ is the weight coefficient, and x i is the visual feature x extracted from the i-th known-class image i , y i is the semantic feature extracted from the class attribute of the i-th known-class image, is the generated feature obtained by the generator according to the semantic feature y i , is the second norm expected by the discriminator D, is the visual feature x i and the generated feature is the interpolation between them.

4. The zero-shot learning method based on visual semantic constraints according to claim 2, wherein In step A22, the loss function of the generator G is: Among them, represents the training loss of the minimization generator G, E represents the mathematical expectation, and y i is the semantic feature extracted according to the class attribute of the i-th known class image, is the generated feature obtained by the generator according to the semantic feature y i , and L vs is the visual semantic constraint between the generated feature of the generator and the known class samples and the unknown class samples, and γ is a hyperparameter.

5. A zero-shot learning generation method based on visual semantic constraints according to any one of claims 1, characterized in that Calculate the visual feature center of the category according to the following formula where p is the number of samples in the i-th class of the known classes, and x j is the visual feature of the j-th image in this class.

6. A zero-shot learning generation method based on visual semantic constraints according to any one of claims 1 to 4, characterized in that In step A3, traverse the semantic features of each unknown-class sample and input them into the trained generator G respectively to generate the visual features of the unknown class Then, the generated visual features of the unknown class are merged with the extracted visual features x of the known class i to obtain the complete visual feature set V of the training data set.

7. A zero-shot learning generation method based on visual semantic constraints according to any one of claims 1 to 4, characterized in that In step A4, the classifier is a Softmax classifier, and when training the classifier, its loss function is: Among them, P n represents the true class label corresponding to the nth visual feature of the visual feature set, Q(c l |v n ) represents the probability that the classifier predicts the class label as c n based on the nth visual feature v l of the visual feature set, and m is the number of samples in the batch training.

Images