Generalized zero-shot learning method based on bidirectional adversarial training and relationship measurement constraint

Abstract

The application relates to the field of image processing and discloses a generalized zero-shot learning method based on bidirectional adversarial training and relationship measurement constraint.1) A data set for zero-shot learning is established;2) visual features of visible class images are extracted by using a pre-trained neural network, and the visual features are preprocessed;3) the visual features extracted in the step 2) are used to train a feature generation network and an attribute generation network through bidirectional adversarial training, and a relationship measurement constraint is added to the training process of the attribute generation network;4) the feature generation network is used to synthesize visual features of invisible classes, the visual features of the visible classes and the invisible classes are fine-tuned through the attribute generation network, and the visual features after the fine-tuning are used to train a classifier through cross-entropy loss. Further fine-tuning of the visual features enables the visual features to be closely related to semantic attributes. The measurement learning mode based on the inter-class relationship effectively improves the feature discriminability and the generalization of the model under the condition of missing invisible classes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a generalized zero-shot learning method based on bidirectional adversarial training and relationship metric constraint which can be used for target recognition, unknown category object recognition, unknown category image retrieval, etc. BACKGROUND

[0002] Zero-shot learning has attracted extensive attention in recent years due to its application prospects and research value, and many excellent algorithms have emerged. These algorithms can be roughly divided into two categories: zero-shot learning methods based on semantic attribute embedding and zero-shot learning methods based on feature generation. Compared with zero-shot learning methods based on semantic embedding, zero-shot learning methods based on feature generation have received more attention. It first trains a visual feature generator conditioned on the semantic attributes of the class, then uses the semantic attributes of the unseen class to synthesize visual features to make up for the lack of unseen class samples in the training stage, and finally trains a classifier using known visual features of the visible class and synthesized visual features of the unseen class. This method only uses one-way adversarial training to synthesize visual features using semantic attributes, which limits the connection between semantic attributes and visual features. Zero-shot learning uses semantic attributes as a bridge for visual knowledge transfer between visible and unseen classes, so the existing generation method does not effectively alleviate the strong bias problem in zero-shot learning. In addition, existing methods ignore the mining of relationships between different classes, which severely restricts the generalization ability of the model, especially in the zero-shot learning setting where the training stage lacks unseen classes.

[0003] Semantic attributes are important clues for knowledge transfer in zero-shot learning. Each class has its corresponding semantic attributes, which are described in the form of vectors and are generally obtained by manual labeling. Each dimension represents the corresponding attribute information, such as specific colors, shapes, etc. Semantic attributes not only contain image description information but also contain a large amount of cross-class information.

[0004] Metric learning is a common method for enhancing the intra-class cohesion and inter-class discriminativeness of instances in current deep learning. It is usually achieved through comparison with positive and negative samples. Through a metric standard (generally the Euclidean distance), the spatial distance of instances in the same class is pulled closer, and the distance between instances of different classes is pulled farther apart. SUMMARY

[0005] The purpose of the present application is to provide a generalized zero-shot learning method based on bidirectional adversarial training and relationship metric constraint for target recognition of unknown categories, which enhances the connection between visual features and semantic attributes while enhancing the intra-class cohesion and inter-class discriminativeness of different classes of visual features, and obtains strongly discriminative features. It can be used for target recognition, unknown category object recognition, unknown category image retrieval, etc.

[0006] The present application comprises the following steps:

[0007] 1) Establishing a data set: establishing a data set for zero-shot learning, introducing semantic attributes as clues for each category;

[0008] 2) Visual feature extraction: using a pre-trained neural network to extract original visual features of visible class images;

[0009] 3) Bidirectional adversarial training and relationship measurement constraint: using the original visual features extracted in step 2) to train the feature generation network and the attribute generation network through bidirectional adversarial training, and simultaneously adding a relationship measurement constraint in the training process of the attribute generation network;

[0010] 4) Training the classifier: using the feature generation network to synthesize visual features of invisible classes, fine-tuning the visual features of visible classes and invisible classes through the attribute generation network, and training the classifier through cross-entropy loss using the fine-tuned visual features.

[0011] In step 1), the data set for zero-shot learning is N represents the number of samples in the data set, x i represents the visual features of the image extracted by the pre-trained backbone neural network, y i represents the corresponding class label, represents the class set of visible classes; for each category, zero-shot learning introduces semantic attributes as clues, where represents the class set of invisible classes, and

[0012] In step 2), the specific steps of visual feature extraction can be: pre-processing all visual features extracted by the backbone network, sampling the data set, taking N1 categories in each batch, selecting N2 different samples for each category, and inputting the corresponding semantic attributes a as the network input;

[0013] In step 3), the specific steps of bidirectional adversarial training of the feature generation network and the attribute generation network can be:

[0014] (1) Combine the semantic attribute a and a randomly sampled Gaussian noise ∈ into the feature generator G F , and the feature generator synthesizes a fake visual feature from the semantic attribute a and the Gaussian noise ∈. Combine the fake visual feature with the real visual feature x and the semantic attribute a, and input them into the feature discriminator D F , and the feature discriminator learns to distinguish (x, a) and and the authenticity of the pair data is learned by the attribute discriminator through an adversarial loss The two similarity matrices are constrained.

[0015] (2) The original visual feature x extracted by the backbone neural network is input into the encoder E to obtain the fine-tuned visual feature h, and the fine-tuned visual feature h is input into the attribute generator GA to synthesize the fake semantic attribute Soft labels are designed through the semantic attribute similarity between categories The fake semantic attribute and the real semantic attribute a are combined and input into the attribute discriminator DA, and the attribute discriminator learns to distinguish and The authenticity of the pair data is learned by the attribute discriminator through an adversarial loss and the attribute generator iteratively cross to complete training; for the fake semantic attribute The constraint is performed through the infoNCE loss to ensure its discriminability in the attribute space.

[0016] Further, the training of the attribute generation network can specifically include the following steps:

[0017] For visible class images, the corresponding original visual feature x is obtained after visual feature extraction; the original visual feature x is fine-tuned by the encoder E to obtain the fine-tuned visual feature h: h=E(x); the fine-tuned visual feature h is input into the attribute generator G A to synthesize the fake attribute The real semantic attribute a corresponding to the fine-tuned visual feature h and the fake semantic attribute are combined with the soft label corresponding to h and input into the attribute discriminator D A The attribute discriminator D A learns to distinguish between real pair data and fake pair data and cross-competes with the attribute generator to complete training, and the adversarial loss function is as follows:

[0018]

[0019] wherein, β~U(0, 1), and η is the weight of the gradient penalty term.

[0020] The soft label ​​is the class label with cross-class information, and is calculated by the similarity of semantic attributes between different classes as follows:

[0021]

[0022] where M represents the number of all classes, represents the soft label of the kth class; represents the semantic attribute (vector) of the ith class, and D represents the dimension of a i .

[0023] In order to ensure the discriminability of the pseudo attribute in the attribute space, the infoNCE loss is used to constrain by contrast learning. The specific loss function is as follows:

[0024]

[0025] where τ represents a temperature hyperparameter, and M is the total number of all class attribute vectors a + . respectively represent the corresponding positive sample and negative sample.

[0026] The training process of the attribute generation network is simultaneously subjected to a relationship measurement constraint, which is a relationship-based measurement learning designed in the training process of the attribute generation network to constrain the fine-tuned visual feature h. The positive sample h + and the negative sample h - of the fine-tuned visual feature h + are randomly sampled, and the similarity probability between h and h - is predicted by a learnable measurement function F. Then, the cosine similarity between the semantic attributes corresponding to the samples is taken as the ground truth to train the whole process by cross-entropy loss. By mining the similarity between semantic attributes, the visual feature is constrained by the relationship between classes, so that the generalization of the model is improved. The specific steps include:

[0027] (1) In order to ensure the discriminability of the fine-tuned feature instances, a relationship-based measurement learning is proposed to constrain them, considering the relationship between different classes and taking the similarity between semantic attributes as the measurement standard between visual feature instances, so as to improve the generalization performance of the model by mining the relationship between classes, thereby facilitating the knowledge transfer from visible classes to invisible classes. Specifically,

[0028] First, a learnable measurement function F is used to predict the similarity probability between two visual features h i and h j . The implementation of the measurement function is as follows:

[0029]

[0030] where w F is a fully connected layer of 2048x1, 2048 is the dimension of h, represents element-wise multiplication, and σ represents a sigmoid activation function; after the measurement function, the similarity between the two visual features h i and h j is expressed in the form of probability; the cosine similarity of the semantic attributes corresponding to the two visual features is calculated as the standard of relationship measurement learning and is calculated through binary cross entropy, and the relationship measurement loss function is as follows:

[0031]

[0032] where cos() represents cosine similarity, a i and a j are the semantic attributes corresponding to h i and h j respectively.

[0033] (2) In the training process of each batch, in order to make the training more sufficient, the positive sample h + and the negative sample h - are randomly sampled for each sample of the fine-tuned visual feature h + , respectively, and the relationship measurement loss is calculated for (h, h + ) and (h, h - ) respectively, and the final loss function is as follows:

[0034]

[0035] Through the relationship measurement loss, the intra-class cohesion and inter-class distinction between visual feature instances will be obviously improved, and the mining of the relationship between classes can greatly improve the generalization ability of the model.

[0036] In step 4), the specific steps of training the classifier can be:

[0037] (1) After the second stage training is iterated T1 times until convergence, the feature generation network and the attribute generation network are effectively trained, and the visual features of the invisible classes are synthesized through the feature generation network using the randomly sampled Gaussian noise and the semantic attributes of the invisible classes as conditions. Meanwhile, for the existing visual features of the visible classes and the synthesized visual features of the invisible classes, the attribute generation network is used for fine-tuning, and the fine-tuning process can be represented as h = E(x). Through the bidirectional adversarial training and the constraint of relationship measurement, the relationship between the visual features and the semantic attributes is greatly improved, which is conducive to the visual knowledge transfer process with semantic attributes as the bridge;

[0038] (2) After obtaining the fine-tuned visual features of the visible class and the invisible class, a fully connected classifier is trained by using the features through cross-entropy loss, and after T2 training iterations, the trained classifier C has the ability to recognize the visible class and the invisible class at the same time;

[0039] (3) In the test stage, the trained classifier C is used to predict the visible class and the invisible class respectively, and the Top-1 recognition accuracy is calculated, which is denoted as S and U respectively, and the harmonic mean of S and U is calculated, which is denoted as H.

[0040] The present application has the following outstanding advantages:

[0041] 1. The present application proposes a bidirectional adversarial training method. Considering the shortcomings of traditional generative methods, an attribute generation network is added to the original feature generation network to further fine-tune the visual features and make the visual features and semantic attributes more closely related. The attribute generation network first fine-tunes the existing visual features, and then inputs the fine-tuned features into the attribute generator as a condition to synthesize semantic attributes. The whole process is completed through adversarial training. The bidirectional adversarial training of attribute to feature and feature to attribute greatly strengthens the relationship between visual features and semantic attributes.

[0042] 2. The present application proposes a new class relationship-based metric learning method, which uses the similarity of semantic attributes between classes to guide the metric learning of visual features. The relationship-based metric constraint is used to complete the inter-instance metric while fully exploiting the relationship between different classes, effectively improving the discriminability of the features and greatly improving the generalization of the model in the case of invisible class missing. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is the bidirectional adversarial training and relationship metric learning framework of the present application.

[0044] Figure 2 is the relationship metric learning explanation of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the present application more clear, the following embodiments will further illustrate the present application with the help of the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. On the contrary, the present application covers any substitution, modification, equivalent method and scheme defined by the claims within the essence and scope of the present application. Further, in order to make the public have a better understanding of the present application, some specific details are described in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0046] The present application comprises the following steps:

[0047] (1) The data set for zero-shot learning is N represents the number of samples in the data set, x i represents the visual features of the image extracted by the pre-trained backbone neural network, y i represents the corresponding class label, represents the class set of visible classes. For each class, zero-shot learning introduces semantic attributes as a clue, where represents the class set of invisible classes, and

[0048] (2) In the first stage, all visual features extracted by the backbone network are preprocessed, the data set is sampled, N1 classes are taken in each batch, N2 different samples are selected for each class, and the corresponding semantic attributes are taken as the input of the network;

[0049] (3) In the second stage, first, the semantic attribute a and a randomly sampled Gaussian noise ∈ are combined and input into the feature generator (G F ), which synthesizes a fake visual feature from the semantic attribute and the Gaussian noise. Then the fake visual feature and the real visual feature x are combined with the semantic attribute a and input into the feature discriminator (D F ), which learns to distinguish between (x, a) and and completes the training through the adversarial loss iterative cross. For the fake visual features, the cosine similarity matrix between the fake visual features in the same batch is calculated, and then the similarity matrix between the corresponding semantic attributes is calculated, and the graph regularization loss is used to constrain the two similarity matrices;

[0050] (4) The visual features x extracted by the backbone neural network are input into the encoder (E) to obtain the fine-tuned features h, and h is input into the attribute generator (GA) to synthesize the fake semantic attribute In addition, in order to enable the model to better learn cross-class knowledge for knowledge transfer, a soft label is designed through the semantic attribute similarity between classes. The fake semantic attribute and the real semantic attribute a are combined with the soft label and input into the attribute discriminator (DA), which learns to distinguish between and The training is iteratively completed by cross-completion with the attribute generator. Meanwhile, the semantic attributes are forged by infoNCE loss Constraints are made to ensure their discriminability in the attribute space.

[0051] (4.1) For visible class images, after step (2) in requirement 1, the corresponding feature x is obtained, which represents the visual feature of the image after preprocessing. First, the visual feature x is fine-tuned by the encoder E to obtain the fine-tuned feature h: h = E(x). Then, the fine-tuned visual feature h is input into the attribute generator (G A ) to synthesize forged attributes h corresponds to the real semantic attribute a and the forged semantic attribute corresponding to h respectively After being combined, they are input into the attribute discriminator (D A ), which learns to distinguish between real pairs of data and forged pairs of data and completes training in cross-antagonism with the attribute generator. The adversarial loss function is as follows:

[0052]

[0053] where β ~ U(0, 1), and η is the weight of the gradient penalty term.

[0054] (4.2) The soft label is a class label designed by the present application with cross-class information, which is different from the traditional hot encoding label containing only single class information. is designed by the similarity of semantic attributes between different classes, and the calculation method is as follows:

[0055]

[0056] wherein, represents the soft label of the kth class. represents the semantic attribute (vector) of the ith class, and D represents the dimension of a i .

[0057] (4.3) To ensure the discriminability of the forged attribute in the attribute space, the infoNCE loss is used to constrain the forged semantic attribute by contrast learning. The specific loss function is as follows:

[0058]

[0059] wherein, τ represents the temperature hyperparameter, M represents the number of all classes, and a+ , respectively represent corresponding positive and negative samples.

[0060] (5) According to the core idea of metric learning, a relationship-based metric learning is designed to constrain the fine-tuned feature h in the attribute generation network training process. The positive samples h + and negative samples h - are randomly sampled. + The similarity probability between h and h - and h and h i is predicted by a learnable metric function (F), and then the cosine similarity between the semantic attributes corresponding to the sample pair is taken as the ground truth to train the whole process by cross-entropy loss. By mining the similarity between semantic attributes, the visual features are constrained by the relationship between classes, so that the generalization of the model is improved.

[0061] (5.1) In order to ensure the discriminability between fine-tuned feature instances, relationship metric learning is proposed to constrain it. Unlike traditional metric learning, which simply pulls the intra-class instance distance and pushes the inter-class instance distance in the feature space, the proposed relationship metric learning fully considers the relationship between different classes and takes the similarity between semantic attributes as the metric standard between visual feature instances. By mining the relationship between classes, the generalization performance of the model can be greatly improved, which is conducive to the knowledge transfer from visible classes to invisible classes. Specifically, first, a learnable metric function (F) is used to predict the similarity probability between two visual features h i and h j The implementation of the metric function is as follows:

[0062]

[0063] where w F is a 2048x1 full connection layer, 2048 is the dimension of h, represents element-wise multiplication, and σ represents the sigmoid activation function; after the metric function, the similarity between the two visual features is represented in the form of probability; at the same time, the cosine similarity of the semantic attributes corresponding to the two visual features is calculated as the standard of relationship metric learning and is calculated by binary cross-entropy. The relationship metric loss function is as follows:

[0064]

[0065] cod() represents the cosine similarity, a i , a j are the semantic attributes corresponding to h i , h j .

[0066] (5.2) In the training process of each batch, in order to make the training more sufficient, the positive sample h of each sample h is randomly sampled respectively + and the negative sample h - , the relationship measurement loss is calculated respectively for (h, h + ) and (h, h - ), and the final loss function is as follows:

[0067]

[0068] Through the relationship measurement loss, the intra-class cohesion degree and the inter-class distinction degree between visual feature instances will be obviously improved, and the mining of the relationship between the classes can greatly improve the generalization ability of the model.

[0069] (6) After the second stage training is iterated T1 times until convergence, the feature generation network and the attribute generation network are effectively trained, the visual features of the invisible classes are synthesized by the feature generation network using the randomly sampled Gaussian noise and the semantic attributes of the invisible classes as conditions, and for the existing visual features of the visible classes and the synthesized visual features of the invisible classes, the attribute generation network is used for fine-tuning, and the fine-tuning process can be represented as h=E(x). Through the bidirectional adversarial training and the constraint of the relationship measurement, the connection between the visual features and the semantic attributes is greatly improved, thereby facilitating the visual knowledge transfer process with the semantic attributes as the bridge;

[0070] (7) After obtaining the fine-tuned visual features of the visible classes and the invisible classes, a fully connected classifier is trained using these features through a cross-entropy loss, and after T2 training iterations, the trained classifier C has the recognition ability for the visible classes and the invisible classes;

[0071] (8) In the test stage, the trained classifier C in step (7) is used to predict the visible class and the invisible class instances respectively and calculate the Top-1 recognition accuracy, which are denoted as S and U respectively, and the harmonic mean of S and U is calculated, denoted as H.

[0072] Referring to Figure 1 , the framework of the present application is:

[0073] Step 1, obtaining the visual features of the visual images and sampling.

[0074] The training sample of the present application is the visual feature, the visual features of the visible class visual images used for training are extracted through the backbone network pre-trained on the ImageNet, the parameters of the backbone network are fixed throughout the training and are not involved in the training, and there is no gradient propagation.

[0075] The visual features are normalized and preprocessed, and then sampled, N1 categories are randomly sampled in each batch, and N2 samples are sampled in each category. While sampling the visual features, the corresponding semantic attributes are also sampled, and the soft labels of each sample are calculated. Each batch contains N1xN2 visual features and their corresponding semantic attributes and soft labels.

[0076] Step 2, training the feature generation network.

[0077] (2a) Combine the semantic attributes with a randomly sampled Gaussian noise and input them into the feature generator (G F ) to obtain the forged visual features. Combine the forged visual features and the corresponding semantic attributes as negative samples, and combine the real visual features and the corresponding semantic attributes as positive samples. Input the positive and negative samples into the feature discriminator (D F ) to train the discriminator. After T i iterations, the feature discriminator parameters are frozen;

[0078] (2b) Repeat the operation in step (2a) to obtain the forged visual features. Combine the forged visual features and the corresponding semantic attributes as positive samples and input them into the feature discriminator (D F ). Then train the feature generator (G F ) through the discriminative loss feedback from the discriminator. At the same time, calculate the cosine similarity matrix of the forged visual features and the cosine similarity matrix of the corresponding semantic attributes in the same batch. Use the graph regularization loss to constrain the training of the feature generator (G F ). After training, the feature discriminator parameters are unfrozen.

[0079] Step 3, training the attribute generation network.

[0080] (3a) Fine-tune the visual features through the encoder (E), and input the fine-tuned features into the attribute generator (G A ) to obtain the forged semantic attributes. Combine the forged semantic attributes with the corresponding soft labels as negative samples, and combine the real semantic attributes with the corresponding soft labels as positive samples. Input the positive and negative samples into the attribute discriminator (D A ) to train the discriminator. After Ti iterations, the attribute discriminator parameters are frozen;

[0081] (3b) Repeat the operation in (3a) to obtain the forged semantic attributes. Combine the forged semantic attributes and the corresponding soft labels as positive samples and input them into the attribute discriminator (D A ). Then train the attribute generator (G A ) through the discriminative loss feedback from the discriminator. At the same time, calculate the cosine similarity matrix of the forged visual features and the cosine similarity matrix of the corresponding semantic attributes in the same batch. Use the graph regularization loss Constraining semantic attributes of forgeries to train attribute generator (G A );

[0082] (3c) Refer to Figure 2 , use the learnable metric function (F) to calculate the similarity probability between fine-tuned features and their positive and negative samples, while calculating the cosine similarity between the corresponding semantic attributes, and use cross-entropy loss to narrow the distance between the two, so as to train the metric function (F) and the encoder (E).

[0083] Experimental results and result analysis:

[0084] Experiment one, using the present application in Animals with Attributes (AWA) dataset for zero-shot target recognition.

[0085] In order to verify the effectiveness of the algorithm, on the test set of AWA, ablation experiment is carried out, and table 1 is the experimental results. Among them, 'BAT' represents bidirectional adversarial training, 'RM' represents relationship metric learning, '√' and '×' represent using and not using the corresponding item method respectively, 'U', 'S' and 'H' represent the average recognition accuracy of visible class and invisible class respectively and the harmonic mean of the two. The experimental results show that the two methods proposed in the present application have a great degree of performance improvement for zero-shot task respectively, which verifies the effectiveness of bidirectional adversarial training and relationship metric learning.

[0086] Experiment two, using the present application in Oxford Flowers (FLO) dataset for zero-shot target recognition.

[0087] In order to verify the effectiveness of the algorithm, on the dataset of FLO, visible class and invisible class are recognized. Table 2 is the experimental results, from which it can be found that the generalized zero-shot learning method based on bidirectional adversarial training and relationship metric constraint proposed in the present application also obtains excellent performance improvement on FLO dataset.

[0088] Table 1

[0089] Index BAT RM U S H 1 × × 57.3 68.9 62.6 2 √ × 62.1 79.0 69.5 3 × √ 63.8 77.7 70.1 4 √ √ 64.8 82.6 72.6

[0090] Table 2

[0091] U S H 69.1 84.0 75.8

[0092] Combining experiment one and experiment two, the present application has significant performance advantage on the existing two zero-shot learning benchmark datasets, surpassing the highest level in the current academic field, verifying the effectiveness of the bidirectional adversarial training and relationship metric learning method proposed in the present application.

Claims

1. A generalized zero-shot learning method based on bidirectional adversarial training and relational metric constraints, characterized in that... Includes the following steps: 1) Create a dataset: Create a dataset for zero-shot learning. For each category, zero-shot learning introduces semantic attributes as cues. 2) Visual feature extraction: Visual features of visible images are extracted using a pre-trained neural network, and the visual features are preprocessed. 3) Bi-directional adversarial training and relational metric constraints: The visual features extracted in step 2) are used to train the feature generation network and the attribute generation network through bi-directional adversarial training. Relational metric constraints are added to the training process of the attribute generation network. The training process of the attribute generation network incorporates relation metric constraints. This involves designing a relation-based metric learning method during network training to constrain the fine-tuned visual features h, and randomly sampling positive samples h of the fine-tuned visual features h. + and negative sample h - Predict h and h using a learnable metric function F + and h and h - The similarity probability between samples is calculated, and then the cosine similarity between the semantic attributes corresponding to the samples is used as the ground truth to train the entire process using cross-entropy loss. By mining the similarity between semantic attributes, visual features are constrained by the relationship between categories, thereby improving the generalization ability of the model. The specific steps include: (1) To ensure the discriminability among fine-tuned feature instances, a relation-based metric learning is proposed to constrain them. It considers the relationships between different categories and uses the similarity between semantic attributes as the metric between visual feature instances. By mining the relationships between categories, the generalization performance of the model is improved, which is conducive to knowledge transfer from visible classes to invisible classes. Specifically, two visual features h are first predicted through a learnable metric function F. i with h j The similarity probability between them is implemented using the following function: Among them, w F It is a 2048×1 fully connected layer, where 2048 is the dimension of h. This indicates element-wise multiplication, where σ represents the sigmoid activation function; after passing through the metric function, the two visual features h i with h j The similarity between them is represented in the form of probability; the cosine similarity of the semantic attributes corresponding to two visual features is calculated as the standard for relation metric learning and is calculated through binary cross-entropy. The relation metric loss function is as follows: Where cos() represents the cosine similarity, a i a j They are h i h j The corresponding semantic attributes; (2) During the training process of each batch, in order to make the training more thorough, positive samples h are randomly sampled for each sample of the fine-tuned visual feature h. + and negative sample h - For (h,h) + ) and (h,h - The relationship metric loss is calculated, and the final loss function is as follows: By using relational measurement loss, the intra-class aggregation and inter-class discrimination of visual feature instances will be significantly improved, while the mining of relationships between categories can greatly improve the generalization ability of the model. 4) Training the classifier: Use a feature generation network to synthesize visual features of the invisible class, fine-tune the visual features of the visible and invisible classes through an attribute generation network, and use the fine-tuned visual features to train the classifier through cross-entropy loss.

2. The generalized zero-shot learning method based on bidirectional adversarial training and relation metric constraints as described in claim 1, characterized in that... In step 1), the dataset for zero-shot learning is N represents the number of samples in the dataset, x i y represents the visual features of an image extracted by a pre-trained backbone neural network. i This indicates its corresponding category label. The set of categories represents the visible classes; for each category, zero-shot learning introduces semantic attributes. As a clue, among them Represents the set of categories for invisible classes, and 3. The generalized zero-shot learning method based on bidirectional adversarial training and relation metric constraints as described in claim 1, characterized in that... In step 2), the specific steps of visual feature extraction are as follows: preprocess all the visual features extracted by the backbone network, sample the dataset, take N1 categories in each batch, select N2 different samples for each category and their corresponding semantic attribute a as the input of the network.

4. The generalized zero-shot learning method based on bidirectional adversarial training and relation metric constraints as described in claim 1, characterized in that... In step 3), the specific steps for training the bidirectional adversarial feature generation network and attribute generation network are as follows: (1) Combine semantic attribute a and a randomly sampled Gaussian noise ε and input them into feature generator G. F In this process, the feature generator synthesizes a fake visual feature based on semantic attribute 'a' and Gaussian noise 'ε'. Fake visual features The true visual feature x is combined with the semantic attribute a and then input into the feature discriminator D. F In the middle, the feature discriminator learns to distinguish between (x,a) and The truth or falsehood and the loss of combat Training is completed iteratively; for forged visual features, the cosine similarity matrix between forged visual features in the same batch is calculated, and the similarity matrix between their corresponding semantic attributes is calculated, using graph regularization loss. Constrain two similarity matrices; (2) Input the real visual features x extracted by the backbone neural network into the encoder E to obtain the fine-tuned visual features h, and input the fine-tuned visual features h into the attribute generator GA to synthesize fake semantic attributes. Design soft tags based on semantic attribute similarity between categories. Fake semantic attributes The true semantic attribute 'a' is compared with the soft tag. The combined input is the attribute discriminator DA, which learns to make discriminations. and The truth or falsehood and the loss of combat Training is completed iteratively by crossing over with the attribute generator; for forged semantic attributes... Loss through infoNCE Constraints are imposed to ensure its discriminability in the attribute space.

5. The generalized zero-shot learning method based on bidirectional adversarial training and relation metric constraints as described in claim 4, characterized in that... The specific steps for training the attribute generation network are as follows: For a visible image, the original visual features x are obtained after visual feature extraction; the original visual features x are fine-tuned by encoder E to obtain the fine-tuned visual features h: h = E(x); the fine-tuned visual features h are then input into the attribute generator G. A Synthesize fake attributes The real semantic attribute a and the fake semantic attribute corresponding to the fine-tuned visual feature h. The soft tags corresponding to h respectively Combined input attribute discriminator D A In the middle, the attribute discriminator D A Learning to distinguish real pairwise data And forged paired data The training is completed through cross-adversarial training with the attribute generator, and the adversarial loss function is as follows: in, β~U(0,1), where η is the weight of the gradient penalty term.

6. The generalized zero-shot learning method based on bidirectional adversarial training and relation metric constraints as described in claim 5, characterized in that... The soft label These are category labels that contain cross-category information. They are designed based on the semantic attribute similarity between different categories, and the calculation method is as follows: Where M represents the total number of categories, This represents the soft label for the k-th category; Let D represent the semantic attribute of the i-th category, and let a represent the semantic attribute of the i-th category. i Dimensions.

7. The generalized zero-shot learning method based on bidirectional adversarial training and relation metric constraints as described in claim 5, characterized in that... To ensure the authenticity of the forged semantic attributes Discriminability in the attribute space is achieved by using infoNCE loss and contrastive learning to constrain the forged semantic attributes. The specific loss function is as follows: Where τ represents the temperature hyperparameter, M is the total number of all categories, and a + , They represent The corresponding positive and negative samples.

8. The generalized zero-shot learning method based on bidirectional adversarial training and relation metric constraints as described in claim 1, characterized in that... In step 4), the specific steps for training the classifier are as follows: (1) After the second stage training iteration T1 times until convergence, both the feature generation network and the attribute generation network are effectively trained. Using randomly sampled Gaussian noise and the semantic attributes of the invisible class as conditions, the visual features of the invisible class are synthesized through the feature generation network. At the same time, for the existing visual features of the visible class and the synthesized visual features of the invisible class, the attribute generation network is used for fine-tuning. The fine-tuning process can be represented as h=E(x). Through bidirectional adversarial training and the constraint of relation measurement, the connection between visual features and semantic attributes is greatly improved, which is conducive to the visual knowledge transfer process with semantic attributes as the bridge. (2) After obtaining the fine-tuned visual features of the visible and invisible classes, a fully connected classifier is trained using these features through cross-entropy loss. After T2 training iterations until convergence, the trained classifier C will have the ability to recognize both the visible and invisible classes. (3) During the testing phase, the trained classifier C is used to predict the visible and invisible instances respectively and calculate their Top-1 recognition accuracy, denoted as S and U respectively. The harmonic mean of S and U is calculated and denoted as H.

Images