Generalized Zero-Shot Image Classification Method Based on Feature Refinement Self-Supervised Learning

Abstract

The present invention is a generalized zero-shot image classification method based on feature refinement self-supervised learning. This method introduces a self-supervised learning task and adds two classification heads to the Swin Transformer network, mainly to address the problem of bias towards visible categories. Through the rotation angle classification task and the contrastive learning task, the visual feature localization is enhanced, and the correlation between visual features and semantic information is strengthened. At the same time, in order to further alleviate the bias problem, pseudo-labels are generated for unseen categories in the training sample construction, thus setting the GZSL task as a transductive learning.

Description

Technical Field

[0001] This application belongs to the field of computer vision, and relates to a generalized zero-shot image classification method, specifically to a generalized zero-shot image classification method based on feature refinement self-supervised learning. Background Art

[0002] Zero-shot Learning (ZSL) refers to image classification when there is no intersection in the data categories between the training set and the test set. Its purpose is to predict and identify data of unseen categories based on data of visible categories, supplemented by relevant common sense information or prior knowledge. The auxiliary information mainly refers to semantic information, including manually defined attribute vectors, text information automatically extracted based on machine learning methods, or combinations thereof. Semantic information can build a bridge between visible and unseen categories.

[0003] In traditional ZSL technology, the test set only contains samples from unseen categories. In the real world, this setting is unreasonable and difficult to achieve. In practical applications, data samples of visible categories are more common than those of unseen categories. We need the test set to contain both unseen category samples and visible category samples. Moreover,

[0004] identifying samples containing two categories simultaneously is more important than only identifying samples of unseen categories. Therefore, to further adapt to the real world, researchers have proposed Generalized Zero-Shot Learning (GZSL), which can identify samples from both visible and unseen categories.

[0005] Most existing GZSL methods focus on embedding-based models and generative-based models. Embedding methods focus on embedding visual features and semantic descriptions into a common space. For example, visual features are mapped to the semantic space, and the similarity between the two modalities is measured. Patent CN 113139591A discloses a generalized zero-shot image classification method based on enhanced multimodal alignment, which also uses the embedding method for alignment. This method uses a hypersphere encoder to construct a latent space for visual features and semantic features, promoting modality alignment. The generative model method first trains a generator to generate visual features of unseen classes, such as Generative Adversarial Network (GAN) or Variational Autoencoders (VAE), and then uses samples of seen classes and samples of unseen classes synthesized by the generator to train a classifier to distinguish different classes. Chinese Patent CN113177587A discloses a generalized zero-shot object classification method based on active learning and variational autoencoders, which significantly improves the classification accuracy of generalized zero-shot. Recently, attention-based methods have become popular because they can directly identify parts of an image related to semantic information, thus capturing the global features and local information of the image. However, attention-based methods still inevitably have the problem of bias towards seen classes.

[0006] Although in the GZSL setting, the label spaces of seen classes and unseen classes have no intersection, there are still overlapping regions between the seen domain and the unseen domain during the model training process, especially when dealing with fine-grained datasets. For example, orcas and humpback whales are seen classes that can be accessed during the training phase, and dolphins are unseen classes for testing. These three species share a large amount of visual features and semantic information. Since the GZSL model only uses seen class data during the training phase, a dolphin sample can easily be misidentified as an orca or a humpback whale. This phenomenon will lead to a decrease in the classification accuracy of unseen classes. Although attention-based models can accurately focus on semantically relevant parts, they still tend to be biased towards seen classes because the model fails to specially process parts of the image features that are irrelevant to semantics. Summary of the Invention

[0007] To overcome the defects and deficiencies of the existing technologies, the present invention provides a generalized zero-shot image classification method based on feature refinement self-supervised learning. This method applies the ShiftedWindows (Swin) Transformer to extract the visual features of images. Different from the existing Swin Transformer encoders, the encoder of this patent is based on self-supervised learning (SSL) tasks. By adding two classification heads (tokens), two self-supervised tasks are introduced respectively: (1) the rotation angle classification task and (2) the contrastive learning task. In addition, a visual feature refinement module and a semantic feature refinement module are constructed to further refine the features. The visual feature refinement module mainly uses the bilinear pooling algorithm to further adapt the visual features to the fine-grained data set and the fine-grained image classification. The semantic feature refinement module mainly strengthens the correlation between the visual features and semantic information through the ImageAdaptive Semantics (IAS), alleviating the bias problem. The core innovations of this application are: first, the Swin Transformer model is improved by adding two tokens to introduce self-supervised learning tasks; second, a new contrastive learning task is proposed, which constrains the visual features obtained by the same image through different feature extractors (ResNet101 and Swin Transformer) to be close enough in the feature space, and the visual features obtained by different images through the same feature extractor to be far enough apart in the feature space.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A generalized zero-shot image classification method based on feature refinement self-supervised learning, the method comprising the following steps:

[0010] (1) Obtain the image data set and the semantic attribute data set of the generalized zero-shot classification model, and use the ResNet101 network to train the semantic attribute data set to obtain the semantic feature A;

[0011] (2) Use the visible class image data set and the semantic attribute data set in the image data set for training to obtain a visible class conditional visual classifier conditional on the semantic attribute. Then use the unseen class image data set and the semantic attribute data set for training to obtain an unseen class conditional visual classifier conditional on the semantic attribute; obtain the weight matrix W of the visible class through the visible class conditional visual classifier s , and use the weight matrix W s of the visible class as the classification weight of the unseen class conditional visual classifier to obtain the pseudo-label of the unseen class image x u Use the obtained pseudo-labels as the labels of unseen classes, and form a new image dataset with the unseen-class images in the image dataset and the pseudo-labels obtained by the conditional visual classifier for unseen classes. Use the image dataset of visible classes, unseen-class images, and pseudo-labels in the image dataset to form a new image dataset as the training samples of the generalized zero-shot classification model for subsequent classification training. The test samples include the image dataset of visible classes and the image dataset of unseen classes with true labels of unseen classes.

[0012] (3) Construct a generalized zero-shot classification model;

[0013] The generalized zero-shot classification model includes a visual feature refinement module, a semantic feature refinement module, a Swin Transformer network with two added classification tokens, and two constructed self-supervised learning tasks. The two newly added classification tokens correspond to two self-supervised learning tasks: the rotation angle classification task and the contrastive learning task. In the rotation angle classification task, the training samples need to be randomly rotated by four different angles, and finally the rotation category is predicted. The contrastive learning task is to input the training samples into the ResNet101 and Swin Transformer networks respectively to obtain visual features extracted by different feature extractors, and be constrained by the contrastive loss function.

[0014] Construct a self-supervised learning module: The self-supervised learning module includes the rotation angle classification task and the contrastive learning task. The rotation angle classification task realizes self-supervised learning through a pre-task of predicting the rotation angle of the image. Randomly rotate the images in the training samples by four different angles (0°, 90°, 180°, and 270°) to obtain a rotated image dataset with rotation category labels. The rotated image dataset is input into the Swin Transformer network to obtain the visual features with rotation information corresponding to the rotated image dataset for predicting their rotation categories. The visual features with rotation angles encoded by the Swin Transformer network are only used to predict the rotation angle of the image and do not directly participate in the subsequent network model of this application, but need to correspond to a classification token in the Swin Transformer network. The contrastive learning task obtains different visual features by inputting the training samples into the ResNet101 and Swin Transformer respectively, and the entire contrastive learning task is constrained by the L NCE oss function.

[0015] After passing through the Swin Transformer encoder, the training samples will be input into the visual feature refinement module and the semantic feature refinement module. The visual feature refinement module uses the idea of bilinear pooling, and the semantic feature refinement module introduces image adaptive features.

[0016] Construct a visual feature refinement module: The visual feature refinement module is improved based on the idea of bilinear pooling and is an improved method for feature fusion. The method adopted in this application is the homologous bilinear pooling method. The input of the visual feature refinement module is the visual feature x obtained after the training samples pass through the Swin Transformer network. The visual feature refinement module can make the visual feature more suitable for the fine-grained dataset, including Hadamard operation, reshape operation, fully connected layer, and normalization layer. The specific operation is to copy the visual feature x for subsequent feature fusion. The initial dimension of x is q. After the reshape operation, the dimensions of the two visual features x are transformed into 1×q and q×1 respectively, named x1 and x2. x1 is decomposed into the parameter matrix U1 and the feature vector x2 is decomposed into the parameter matrix U2 and the feature vector The parameter matrix U1 and the feature vector The product of is x1, and the feature vector and the product of the parameter matrix U2 is x2. Then U1 and U2 are input into the global vector layer through the Hadamard operation, and are also input into the global vector layer through the Hadamard operation and aggregated into the global vector z in the global vector layer. Then, through a fully connected layer and a normalization operation, the output is the refined visual feature

[0017] Construct a semantic feature refinement module: The semantic feature refinement module is called the ImageAdaptive Semantics (IAS) module. The IAS module can combine the original semantic features for inter-class discrimination with the image-specific attention vectors for intra-class variations, and then map the image-adaptive semantic features into the corresponding visual space, thereby improving the accuracy of GZSL image classification. The IAS module includes the first fully connected layer FC, the first classification function softmax, Hadamard operation, sum operation, the second fully connected layer, the third fully connected layer FC, the first normalization layer, the second normalization layer, and the second classification function softmax. The input of the IAS module is the visual feature output by the visual feature refinement module and the semantic feature A obtained by training the semantic attribute dataset using the ResNet101 network. The visual feature After being processed by the first fully connected layer FC and the first classification function softmax, the result performs Hadamard operation and sum operation with the semantic feature to obtain the improved semantic feature The improved semantic feature After being processed by the second fully connected layer FC, the third fully connected layer FC, and the second normalization layer, the Hadamard operation is performed with the result of the visual feature x after being processed by the second normalization layer, and the second classification function softmax is processed to map the semantic feature into the visual space;

[0018] Using the training samples as the input to the Swin Transformer network, passing through the visual feature refinement module and the IAS module in sequence, outputting the classification categories of the generalized zero-shot image classification task, training the generalized zero-shot classification model, and the total loss function of the generalized zero-shot classification model is the sum of the loss functions of self-supervised learning (the loss function of the rotation angle classification task and the loss function of the contrastive learning task) and the loss of the generalized zero-shot classification task, and is expressed by the formula:

[0019]

[0020] where, is the total loss function of the generalized zero-shot classification model, is the loss function of the rotation angle classification task, is the loss function of the generalized zero-shot classification task, is the loss function of the contrastive learning task;

[0021] The loss function of the generalized zero-shot classification task is:

[0022]

[0023] where, M represents the number of training samples, y i and represent the true label and the predicted label of the generalized zero-shot image classification task respectively;

[0024] The loss function of the rotation angle classification task is:

[0025]

[0026] where, represents the predicted label of the rotation angle classification task, and a ∈ {0, 1, 2, 3} represents 4 rotation angles;

[0027] The loss function of the contrastive learning task is:

[0028]

[0029] where, M represents the number of training samples, x j , respectively represent the visual features of the training samples after passing through the ResNet101 network and the Swin Transformer network, and W represents the weight matrix of the Swin Transformer network. represents x j , the similarity between.

[0030] Thus, a trained generalized zero-shot classification model is obtained.

[0031] (4) Use the trained generalized zero-shot classification model to identify generalized zero-shot images and complete the generalized zero-shot classification task; input the unseen-class images with known semantic features into the trained generalized zero-shot classification model to obtain the predicted labels of the unseen-class images.

[0032] The step (1) of obtaining the image dataset and the semantic attribute dataset is specifically as follows:

[0033] (1.1) Use the existing datasets for image classification and fine-grained image classification as the image dataset, including: Animal with Attribute 2 (AWA2), Caltech-UCSD-Birds-200-2011 (CUB), and SUN Attribute Dataset (SUN). Among them, the CUB and SUN datasets are fine-grained datasets, and AWA2 is a coarse-grained dataset. The dataset AWA2 contains 50 classes of 30,745 and 37,322 animal pictures, where 40 classes are used as training classes and 10 classes are used as test classes; the dataset CUB is a fine-grained dataset of bird pictures, containing 200 species of birds with a total of 11,788 pictures, where 150 classes are used as training classes and 50 classes are used as test classes; the dataset SUN covers a fine-grained dataset of various environmental scenes and internal images, containing 717 classes with a total of 14,340 pictures, where 645 classes are used as training classes and 72 classes are used as test classes. To meet the input requirements of the backbone network, that is, the Swin Transformer network, the image resolution is unified to 224×224, and the visual features X s of the visible-class images and their labels Y s are denoted as D s ={X s , Y s}, as the image dataset of the visible classes, and the visual features X u of the unseen-class images and their labels Y u are denoted as D u ={X u , Y u}, as the image dataset of the unseen classes.

[0034] (1.2) Generally speaking, zero-shot classification requires the use of auxiliary semantic information to construct a semantic space and establish the interaction relationship between the visual modality and the semantic modality. Among them, the auxiliary semantic information can be divided into two categories: auxiliary information based on artificial definition and auxiliary information based on learning. Among them, the auxiliary information based on artificial definition obtains semantic features of different dimensions through the ResNet101 network. The semantic feature A obtained from the semantic attribute dataset of this application through the ResNet101 network is expressed as A = A s ∪A u , where the subscript s represents the visible class and u represents the unseen class. Among them, the AWA2 dataset uses 85-dimensional semantic features, the CUB dataset uses 312-dimensional semantic features, and the SUN dataset uses 102-dimensional semantic features.

[0035] In step (2), in order to further alleviate the problem of bias towards the visible class, this application redefines the generalized zero-shot image classification as a visual classification problem conditional on semantic attributes. The image dataset of the visible class and the semantic attribute dataset are used to train a conditional visual classifier of the visible class conditional on semantic attributes. The image dataset of the unseen class and the semantic attribute dataset are trained to obtain a conditional visual classifier of the unseen class conditional on semantic attributes. The classification weight W s of the conditional visual classifier of the visible class is used as the weight of the conditional visual classifier of the unseen class to obtain a pseudo-label. The pseudo-label and X u constitute new training data and are used as training samples to be input into the SwinTransformer network to participate in the training process, thereby converting the inductive generalized zero-shot method into a transductive setting.

[0036] Step (3) constructs a generalized zero-shot classification model, which is specifically as follows:

[0037] (3.1) SwinTransformer network with two classification tokens added: Using the SwinTransformer network as the backbone network, this application creatively applies the SwinTransformer network to the generalized zero-shot image classification problem, which is an important attempt. This network has achieved powerful performance in recognition tasks such as image classification, object detection, and semantic segmentation. After the images in the training samples are input into the SwinTransformer network, they can become a series of flattened 2D patches x p , and the resolution of each patch is p×p. In order to match the dimensions of the Transformer, the dimension of each patch is set to 48.

[0038] After passing through 4 stages, the flattened patch blocks can obtain the visual feature x of the input image. Each stage contains a patch merging layer and a Swin Transformer block. The patch merging layer is similar to a pooling operation, and the Swin Transformer block includes a window multi-head self-attention module and a shifted window multi-head self-attention module.

[0039] (3.2) Two self-supervised task mechanisms:

[0040] Rotation angle classification task: To achieve self-supervised learning, a task of predicting the rotation angle of an image is constructed. Specifically, the images in the training samples are randomly rotated by four different angles (0°, 90°, 180°, and 270°). The training samples are further processed to obtain a rotated image dataset. Four categories are set for the four rotated angles, and the labels are set as (0, 1, 2, 3). The rotated image dataset is also input into the Swin Transformer network to obtain the visual features with rotation information corresponding to the rotated image dataset, which are used to predict their rotation categories. The process of solving this task does not involve class attributes or semantic information, so the semantic-irrelevant parts in the visual features can be removed.

[0041] Contrastive learning task: The training samples are respectively input into the ResNet101 network and the Swin Transformer network to obtain the visual features of the same image after different feature extractors. These two visual features processed by different feature extractors are used as positive sample pairs. The similarity between the positive sample pairs is calculated through cosine similarity, and the distance between the positive sample pairs is constrained to be close enough through contrastive loss.

[0042] (3.3) Construct a visual feature refinement module: To make the visual features obtained by the Swin Transformer network better adapt to the fine-grained dataset, a visual feature refinement module is constructed in the GZSL method in this application. The visual features after feature fusion are obtained by improving the homologous bilinear pooling method. The input of this module is the visual feature x processed in step (3.1), and the output is the refined visual feature Input into the IAS module.

[0043] (3.4) Construct the IAS module: To enrich the diversity of semantic features and better distinguish the fine-grained dataset, an IAS module is constructed in the GZSL method in this application. It can not only better distinguish different categories but also reflect the visual gap within the class. The input of the IAS module is the refined visual feature processed in step (3.3) and the semantic feature A processed by the ResNet101 network, and the output is the mapping of the above visual feature x and semantic feature A in the visual space.

[0044] (3.5) Input the training samples and the rotated image dataset into the Swin Transformer network simultaneously for feature extraction to obtain their respective corresponding visual features x. Use the obtained visual features to perform the generalized zero-shot image classification task and the rotation angle classification task respectively. Steps (3.1) and (3.2) are jointly trained and share the same Swin Transformer network; train the generalized zero-shot classification model, and the total loss function of the generalized zero-shot classification model is the sum of the loss function of self-supervised learning and the loss of the generalized zero-shot classification task;

[0045] The Swin Transformer network is trained together with the visual feature refinement module and the IAS module. Initialize the number of iterations as k, the maximum number of iterations as K, K≥30, and let k = 1;

[0046] Obtain the trained generalized zero-shot classification model.

[0047] (4) The classification model identifies the generalized zero-shot images to complete the generalized zero-shot classification task. Use the images in the test samples as the input of step (3.1) for feature extraction, and then input the visual features and semantic features A obtained in step (3.1) into the feature refinement module to obtain the mapping of the visual features and semantic features in the visual space. Finally, the probability of each class can be obtained in the visual space to obtain the final classification result.

[0048] The present invention also protects a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the generalized zero-shot image classification method can be implemented.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] (1) The present invention proposes a new generalized zero-shot image classification method, which introduces a self-supervised learning task and adds two classification heads to the Swin Transformer network, mainly used to solve the problem of bias towards visible categories. Through the rotation angle classification task and the contrast learning task, the visual feature localization is enhanced, and the correlation between the visual features and semantic information is strengthened. At the same time, in order to further alleviate the bias problem, pseudo-labels are generated for unseen categories in the construction of training samples, so as to set the GZSL task as a transductive learning.

[0051] (2) The method of the present invention realizes the refinement of visual features and semantic features. The refinement of visual features is mainly achieved by applying the bilinear pooling method. The refinement of semantic features generates specific semantic features for each image, introducing the concept of using image-adaptive features to represent each class. These features can better distinguish the semantic features of samples within a class, thereby improving the accuracy of the final image classification and avoiding single semantic feature description.

[0052] (3) The method of the present invention shows excellent performance on different data sets, greatly improving the accuracy of generalized zero-shot recognition, indicating that the neural network constructed in this application has stronger expression ability and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is the flowchart of the generalized zero-shot image classification method and system of the present invention.

[0054] Figure 2 is the overall network structure diagram of the generalized zero-shot classification model of the present invention.

[0055] Figure 3 is the schematic diagram of the overall network structure of the Swin Transformer of the present invention.

[0056] Figure 4 is the schematic diagram of the network structure of the Swin Transformer block of the present invention.

[0057] Figure 5 is the schematic diagram of the network structure of the visual feature refinement module of the present invention.

[0058] Figure 6 is the schematic diagram of the network structure of the IAS module of the present invention.

[0059] Figure 7 is the schematic diagram of the structure of the transductive setting of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] The technical solutions of the present invention will be further described in detail below in conjunction with specific embodiments and drawings, but the protection scope of this application is not limited thereby.

[0061] The generalized zero-shot image classification method based on feature refinement self-supervised learning of the present invention includes the following steps:

[0062] (1) Obtain the image data set and semantic attribute data set of the generalized zero-shot classification model, and use the ResNet101 network to train the semantic attribute data set to obtain semantic feature A;

[0063] (2) Construct a conditional visual classifier with the image dataset and semantic attribute dataset as inputs, generate pseudo-labels for images of unseen classes, and use the obtained pseudo-labels as the labels of unseen classes. Combine the unseen-class images in the image dataset and the pseudo-labels obtained through the conditional visual classifier to form a new image dataset. Use the image datasets of visible classes, unseen-class images, and pseudo-labels in the image dataset to form a new image dataset as the training samples of the generalized zero-shot classification model for subsequent classification training;

[0064] (3) Construct a generalized zero-shot classification model;

[0065] The generalized zero-shot classification model includes a Swin Transformer network, a visual feature refinement module, and a semantic feature refinement module. The visual feature refinement module is implemented by introducing the bilinear pooling method. The semantic feature refinement module combines the original semantic features for class discrimination with the image-specific attention vectors for intra-class variation through an image adaptation module, and then maps the image-adaptive semantic features into the corresponding visual space to achieve GZSL image classification;

[0066] Training process of the generalized zero-shot classification model:

[0067] Construct a self-supervised module: The self-supervised module includes a rotation angle classification task and a contrastive learning task. In the rotation angle classification task, the images in the training samples are randomly rotated by four different angles to obtain a rotated image dataset with rotation class labels. The rotated image dataset is input into the Swin Transformer network (dual input) to obtain the visual features with rotation information corresponding to the rotated image dataset for predicting their rotation classes;

[0068] The contrastive learning task is to update the parameters of the Swin Transformer network by constraining the distance between positive sample pairs obtained from the same training sample image through the ResNet101 network and the Swin Transformer network with a contrastive loss.

[0069] Input the training samples and the rotated image dataset into the Swin Transformer network, and then input the visual features x obtained from the training samples into the visual feature refinement module and the IAS module to output the classification categories of the generalized zero-shot image classification task, and train the generalized zero-shot classification model. The total loss function of the generalized zero-shot classification model is the sum of the loss function of self-supervised learning and the loss of the generalized zero-shot classification task, and is expressed by the formula:

[0070]

[0071] where, is the total loss function of the generalized zero-shot classification model, is the loss function for the rotation angle classification task, is the loss function for the generalized zero-shot classification task, is the loss function for the contrastive learning task.

[0072] The loss function for the generalized zero-shot classification task is:

[0073]

[0074] where M represents the number of training samples, y i and represent the true label and the predicted label of the generalized zero-shot image classification task, respectively;

[0075] The loss function of self-supervised learning is:

[0076]

[0077] where, represents the predicted label of the rotation angle classification task, a ∈ {0, 1, 2, 3};

[0078] The loss function of the contrastive learning task is:

[0079]

[0080] where M represents the number of training samples, x j , represent the visual features of the training samples after passing through the ResNet101 network and the Swin Transformer network, respectively. W represents the weight matrix of the Swin Transformer network, represents x j , the similarity between them.

[0081] So far, the trained generalized zero-shot classification model is obtained;

[0082] (4) Use the trained generalized zero-shot classification model to identify the generalized zero-shot images and complete the generalized zero-shot classification task.

[0083] The process of obtaining the pseudo-label is as follows: Use the image dataset of visible classes and the semantic attribute dataset in the image dataset for training to obtain a visible class conditional visual classifier conditional on semantic attributes. Then use the image dataset of unseen classes and the semantic attribute dataset for training to obtain an unseen class conditional visual classifier conditional on semantic attributes; Obtain the weight matrix W of the visible classes through the visible class conditional visual classifier s , with the weight matrix W of the visible classess As the classification weight of the conditional visual classifier for unseen classes, obtain the unseen-class image x u with its pseudo-label

[0084] Figure 2 is the overall network structure diagram of the generalized zero-shot classification model of the present invention. The generalized zero-shot classification model of the present application consists of a Swin Transformer network and a feature refinement module, corresponding to three inputs respectively, including two image inputs and one semantic attribute input. The training samples of the GZSL dataset pass through the Swin Transformer network to obtain visual features x, and the generalized zero-shot classification task prediction is performed through the MSE loss. The rotated image dataset is obtained by randomly rotating the original image dataset by four angles. The rotated image dataset is also input into the Swin Transformer network to obtain its corresponding visual features, realizing the pre-task of predicting the rotation angle, which is used to train together with the generalized zero-shot classification task prediction. At the same time, a contrastive learning task is introduced to jointly train the Swin Transformer network. Then, the visual features x obtained from the original image dataset are passed through the visual feature refinement module to obtain refined visual features, and then the refined visual features and semantic features A are input into the IAS module. The IAS module can perform image-adaptive attention supplementation on the semantic features, expanding the semantic features of each class.

[0085] Figure 3 This is the Swin Transformer network structure of the present invention. The input is the resized original dataset. After being segmented into patch blocks, it is sequentially input into 4 stages through a linear embedding layer. Each stage contains a patch merging layer and a Swin Transformer block, and finally the visual features corresponding to the training samples are obtained.

[0086] Figure 4 This is the network structure schematic diagram of the Swin Transformer block. The Swin Transformer block includes a window multi-head self-attention module and a shifted window multi-head self-attention module. The window multi-head self-attention module includes window multi-head self-attention W-MSA operations and MLP operations. Before each operation, it has to go through a Norm layer. There is a residual connection between the two operations. The shifted window multi-head self-attention module is similar to the window multi-head self-attention module, including shifted window multi-head self-attention SW-MSA operations and MLP operations.

[0087] Figure 5It is a schematic diagram of the network structure of the visual feature refinement module of the present invention. The input of the visual feature refinement module is the visual feature x obtained after the training samples pass through the Swin Transformer network. The visual feature refinement module can make the visual feature more suitable for the fine-grained data set, including Hadamard operation, reshape operation, fully connected layer and normalization layer. The specific operation is to copy the visual feature x for subsequent feature fusion. The initial dimension of x is q. After the reshape operation, the dimensions of the two feature vectors x are changed to 1×q and q×1 respectively, named x1 and x2. x1 is decomposed into the parameter matrix U1 and the feature vector x2 is decomposed into the parameter matrix U2 and the feature vector The product of the parameter matrix U1 and the feature vector is x1, and the product of the feature vector and the parameter matrix U2 is x2. Then U1 and U2 are input into the global vector layer through the Hadamard operation, and are also input into the global vector layer through the Hadamard operation and aggregated into the global vector z in the global vector layer. Then, through a fully connected layer and a normalization operation, the output is the refined visual feature

[0088] Figure 6 It is a schematic diagram of the network structure of the IAS module of the present invention. The input of the IAS module is the refined visual feature obtained by the visual feature refinement module and the semantic feature A extracted through the Resnet101 network from the manually annotated semantic attribute data set. The specific semantic feature corresponding to each image is obtained through the self-attention mechanism. Specifically, it includes the first fully connected layer FC, the first classification function softmax, Hadamard operation, sum operation, the second fully connected layer, the third fully connected layer FC, the first normalization layer, the second normalization layer, and the second classification function softmax. The refined visual feature After being processed by the first fully connected layer FC and the first classification function softmax, the result performs the Hadamard operation and sum operation with the semantic feature to obtain the improved semantic feature The improved semantic feature After being processed by the second fully connected layer FC, the third fully connected layer FC and the first normalization layer, it performs the Hadamard operation and the second classification function softmax processing with the result of the visual feature x processed by the second normalization layer, and maps the semantic feature to the visual space.

[0089] Figure 7 It is the structure of the transductive setting of the present invention. Using the visible class semantic feature A s as a condition to train the conditional visual classifier of the visible class to obtain the classification weight W s, and then bring the obtained classification weights into the conditional visual classifier for unseen classes to obtain the label Y of the unseen classes u .

[0090] As Figure 1 shown, the present invention proposes a generalized zero-shot image classification method based on feature refinement self-supervised learning, and the steps are as follows:

[0091] Step 1: Obtain the image dataset and semantic attribute dataset of the generalized zero-shot classification model;

[0092] Obtain the image dataset D containing visible classes from the AWA2, CUB, and SUN datasets s ={X s ,Y s}, where the AWA2 dataset includes 23,547 visible class images, the CUB dataset includes 7,057 visible class images, and the SUN dataset includes 10,320 visible class images. At the same time, select the image dataset D of unseen classes from different datasets u ={X u ,Y u}, and the AWA2, CUB, and SUN datasets contain 7,913, 2,967, and 1,440 unseen class images respectively. The visible classes and unseen classes corresponding to different datasets are also different. The AWA2 dataset includes 40 visible classes and 10 unseen classes, the CUB dataset includes 150 visible classes and 50 unseen classes, and the SUN dataset includes 645 visible classes and 72 unseen classes. After dividing the visible classes and unseen classes, preprocess all the images and crop the image samples into a unified size. The image resolution specified in this application is 224×224. The semantic feature A dimensions of each dataset are also different. Among them, the AWA2 dataset uses 85-dimensional semantic features, the CUB dataset uses 312-dimensional semantic features, and the SUN dataset uses 102-dimensional semantic features.

[0093] Step 2: Training samples:

[0094] As Figure 7 shown, construct a conditional visual classifier with the image dataset and semantic attribute dataset as inputs, convert the GZSL problem into a new conditional visual classification problem, generate pseudo-labels for the images of unseen classes, and thus convert the inductive zero-shot learning into a transductive zero-shot learning method. The input part is the image dataset D of visible classes s ={X s ,Y s} and the semantic attribute dataset A = A s ∪A u ,

[0095] W = f(A) (5)

[0096] Among them, W represents the weight matrix, f represents the relationship between the semantic attribute dataset and the weight matrix in the conditional visual classifier. The image dataset D of the visible class s ={X s ,Y s} and the semantic attribute dataset are input, and the weight matrix W of the visible class is obtained through formula (5) s . At the same time, the unseen class dataset D s ={X s ,Y s} is input into the conditional visual classifier to obtain the conditional visual classifier of the unseen class. Then, the weight matrix W of the visible class s is substituted into the conditional visual classifier of the unseen class to obtain the pseudo-label of the unseen class image X u . Calculate the classification score S of X u according to formula (6) to obtain the pseudo-label of the unseen class image.

[0097]

[0098] Among them, σ is a learnable scalar, W i is the conditional visual classifier weight vector of the i-th class, Num is the total number of classes, and ρ is the probability. To avoid existing noise, a filtering strategy is proposed to reduce the impact of noise. Let be the classification score of the unseen class image u i ∈X u . At the same time, let and be the first and second highest scores. The pseudo-label assigned to u i should satisfy formula (7):

[0099]

[0100] Among them, γ is the threshold for controlling the peak. This constraint is to prevent incorrect label assignment in the conditional visual classifier. In addition, a loss function is provided for the conditional visual classifier task:

[0101]

[0102] Among them, is the probability that x u belongs to y u , and η is a hyperparameter. When the noise level is high, a higher value is preferably selected. is the classification task, and the loss function can use the generated pseudo-labels to update the conditional visual classifier during backpropagation.

[0103] Combine the pseudo - labels and X u to form new training data, which is used as training samples and input into the Swin Transformer network to participate in the training process, thus converting the inductive generalized zero - shot method into a transductive setting.

[0104] As Figure 2 shown, combine the image datasets of visible classes, unseen - class images, and pseudo - labels in the image dataset to form a new image dataset as the training samples of the generalized zero - shot classification model. The images in the training samples are input into the Swin Transformer network to obtain the visual feature x corresponding to the images.

[0105] Step 3: Construct a generalized zero - shot classification model;

[0106] Step 3.1, as Figure 3 shown, to conform to the structure of the Transformer, turn the images in the training samples into a series of flattened 2D patches In this application, C = 3 and P = 4. That is, the images of the training samples are split into patch blocks, and the feature dimension after flattening each patch block is 4×4×3 = 48. The image resolution of the training samples is (H, W), and N is the number of patches, which is also the effective sequence length finally fed into the Transformer. The patch blocks can obtain corresponding linear embeddings through a fully - connected layer, and then these linear embeddings are input into several Swin Transformer blocks with improved self - attention. Compared with the Vision Transformer (ViT), the Swin Transformer is a hierarchical structure with a process of gradually decreasing resolution, which are 4 - fold, 8 - fold, and 16 - fold downsamplings respectively, while the ViT always maintains a 16 - fold downsampling.

[0107] Step 3.2, as Figure 3 shown, to achieve a hierarchical structure, the network architecture of the Swin Transformer is divided into 4 stages. After each stage, the resolution of the patch blocks is halved, and the number of channels becomes twice the original. Each stage includes two parts: a patch merging layer and a Swin Transformer block. The patch merging layer is an operation similar to pooling but without losing information. The Swin Transformer block is similar to the Transformer block, except that the original multi - head self - attention mechanism is replaced by a window multi - head self - attention mechanism and a shifted window multi - head self - attention mechanism.

[0108] Step 3.3, as Figure 4As shown, the Swin Transformer block includes a window multi-head self-attention module and a shifted window multi-head self-attention module. The linear embedding of the patch block is input into the window multi-head self-attention module of the Swin Transformer block after passing through the patch merging layer. The window multi-head self-attention module first includes a Norm layer and then undergoes a window multi-head self-attention operation. After a residual connection, it passes through another Norm layer and an MLP (multi-layer perceptron) operation, and then is input into another shifted window multi-head self-attention module. The shifted window multi-head self-attention module also first includes a Norm layer and then undergoes a shifted window multi-head self-attention operation. After a residual connection, it passes through another Norm layer and an MLP operation. Each stage contains one Swin Transformer block as described above. The calculation formula is as follows:

[0109]

[0110] Among them, W-MSA and SW-MSA represent the window multi-head self-attention operation and the shifted window multi-head self-attention operation respectively, and z l-1 represents the visual feature obtained by the linear embedding corresponding to the patch block in step 3.1 passing through the patch merging layer in step 3.2, represents the visual feature after the window multi-head self-attention operation, and z l represents the output feature after the window multi-head self-attention module, and is also the input feature of the shifted window multi-head self-attention module, represents the visual feature after the shifted window multi-head self-attention operation, and z l+1 represents the visual feature after the entire Swin Transformer block.

[0111] Step 3.4, as Figure 4As shown in the figure, two self-supervised learning tasks are designed in this application. One is to obtain implicit visual features through the rotation angle classification task, and this task corresponds to a classification token of the Swin Transformer network. We randomly rotate the image by 0°, 90°, 180°, and 270°, and then record the rotation labels. The Swin Transformer network is trained to correctly identify the rotated images. Let g(·∣a) be the operator that rotates an image by 90°×a, where a ∈ {0, 1, 2, 3}. The input of this task is the visual features obtained by the Swin Transformer network from the rotated image dataset, which is used for the rotation angle classification task and does not directly participate in the generalized zero-shot image classification task. However, this task and the generalized zero-shot image classification task are trained together, so the two tasks share a Swin Transformer network, which can indirectly affect the final generalized zero-shot image classification task. Further, the process of solving this pre-task does not involve category attributes or semantic information, so the decorrelation operation of visual features can be achieved.

[0112] The second is to further train the parameters of the Swin Transformer network through the contrastive learning task. By inputting the training samples into the ResNet101 network, positive sample pairs for contrastive learning with the visual features obtained by the Swin Transformer network are formed. The parameters of the Swin Transformer network are trained through the contrastive loss, and this process is achieved by training to minimize the distance between positive sample pairs. This application selects cosine similarity as the standard for measuring the similarity of positive sample pairs.

[0113] As Figure 3 shown, by inputting the training samples into the Swin Transformer network, the corresponding visual features can be obtained. The two newly added classification tokens in the Swin Transformer network are only used to solve the rotation angle classification task and the contrastive learning task, and the parameters of the Swin Transformer network are updated by solving these two self-supervised tasks, but do not participate in the generalized zero-shot image classification task.

[0114] Step 3.5, the visual features obtained after the training samples pass through the Swin Transformer network in Step 3.4 are input into the visual feature refinement module. As Figure 5 shown, the feature refinement improves the method of homologous bilinear pooling. The original formula of the method of homologous bilinear pooling is expressed as

[0115]

[0116] Among them, z represents the global vector after feature fusion, and x s and respectively represent two feature vectors participating in feature fusion. The two feature vectors of homogenous bilinear pooling come from the same feature extractor. The dimensionality of the features after traditional bilinear pooling fusion = the product of the dimensionalities of feature x and feature y, which has the risk of overly high dimensionality. In this application, the feature vectors will be further decomposed into a learnable parameter matrix U (with a smaller dimension) and a feature vector (with a larger dimension), and the fused feature vector is input into a fully connected layer for dimensionality conversion, alleviating the problem of overly high dimensionality. Equation (13) can be rewritten as:

[0117]

[0118] Among them, represents the Hadamard operation, and U1 are the vectors after the decomposition of x1, and U2 are the vectors after the decomposition of vector x2. x1 and x2 respectively represent the feature vectors obtained after different reshape operations on the visual feature x obtained by the Swin Transformer network. LN represents the fully connected layer and normalization operation, represents the refined visual feature output by the visual feature refinement module.

[0119] Step 3.6, as Figure 6 shown, guided by the visual feature, that is, the refined visual feature obtained through the visual feature refinement module and the self-attention of semantic feature A, so as to achieve the purpose of adding different attentions to the semantic feature through the IAS module. First, the semantic feature A of each dataset is obtained through the Resnet101 network. This semantic feature serves as the original semantic feature. The original semantic feature can only learn a single embedding vector. After inputting the original semantic feature A into the IAS module, the improved semantic feature is obtained. The specific calculation process is as follows:

[0120]

[0121] Among them, softmax(g(x)) represents the semantic attention predicted through a linear layer g, represents the Hadamard operation. Through this operation, the attention and the semantic feature are combined together. Finally, after two fully connected layer normalization layers, the semantic feature is mapped to the visual space. The whole process can be expressed as:

[0122]

[0123] Among them, A represents the original semantic feature, represents the improved semantic feature, and V(x) represents the corresponding semantic feature.

[0124] Step 3.7: Train the generalized zero-shot classification model, iteratively update it according to the gradient descent method, thereby determining the parameters of the generalized zero-shot classification model, update the weight matrix according to the mean square error loss function, cross-entropy loss function, and contrastive loss function, and obtain the trained generalized zero-shot classification model.

[0125] Step 4: Identify the generalized zero-shot image through the trained generalized zero-shot classification model to complete the image classification task of the generalized zero-shot.

[0126] The network of this application was pre-trained on ImageNet and fine-tuned on the dataset used for experiments. ResNet-101 uses a 2048-dimensional final pooling layer. The patch size in the Swin Transformer network is set to 4×4, with 24 hidden layers, each hidden layer being 1024-dimensional, each layer having 16 heads and 24 series decoders. Our model was trained using the Adam optimizer with a fixed learning rate of 0.0001, and the model can finally converge. All methods were implemented on PyTorch. This application selects the class-average accuracy Macc as the evaluation metric for zero-shot classification, and the calculation formula is

[0127]

[0128] Among them, K u represents the total number of unseen classes, y ts represents the unseen class, represents the classification accuracy of the i-th class in the unseen classes, and nu represents the number of samples in the unseen classes. For the generalized zero-shot classification task, since the search space during testing is not only limited to the unseen classes but also includes the seen classes. A more unified classification standard is to propose a zero-shot performance evaluation metric called harmonic mean accuracy based on the class-average accuracy, and its calculation formula is:

[0129]

[0130] Among them, and They respectively represent the class average accuracies obtained on the image datasets of the visible classes and the unseen classes. The results of this application are verified on the AWA2, CUB, and SUN datasets, and experimental verifications are carried out on network models such as Region Graph Embedding Network (RGEN), Over-Complete Distribution-Conditional Variational Autoencoders (OCD-CVAE), and CrossAttribute-Guided Transformer for Zero-Shot Learning (TransZero++) and the method of this application, and the experimental comparison results shown in Table 1 are obtained.

[0131] Table 1 Experimental comparison results of different methods

[0132]

[0133] As shown in Table 1, S represents the classification accuracy of the visible classes, U represents the classification accuracy of the unseen classes, and H represents the harmonic mean of S and U, which is also the final evaluation index for the generalized zero-shot image classification task. The experimental results show that the method of this application achieves the best harmonic mean on all three datasets, that is, it can achieve the best classification effect on all three datasets. This is because this application has a significant effect on improving the classification accuracy of the unseen classes, and also because our model is effective in alleviating the bias problem.

[0134] In summary, the present invention proposes a method based on feature refinement self-supervised learning to solve the GZSL task. Self-supervised learning is realized by adding two classification tokens to the Swin Transformer network corresponding to two self-supervised tasks respectively. Through visual feature refinement and semantic feature refinement, the visual features are further adapted to fine-grained image classification and semantic decorrelation operations are performed on the visual features. By redefining the GZSL task as a visual classifier task conditional on semantic attributes to generate pseudo-labels for unseen class samples, the inductive generalized zero-shot learning is converted into a transductive setting, further alleviating the bias problem towards the visible classes. Finally, the experiments on three popular datasets also demonstrate the superiority of the model of this application.

[0135] The above embodiments are one application of the present invention, but the implementation schemes of the present invention are not limited thereto. Any other technical changes without departing from the principle and spirit of the present invention are included in the protection scope of the present invention.

[0136] Matters not described in the present invention are applicable to the prior art.

Claims

1. A generalized zero-shot image classification method based on feature refinement self-supervised learning, the method comprising the following steps: (1) Obtain the image dataset and semantic attribute dataset of the generalized zero-shot classification model, and use the ResNet101 network to train the semantic attribute dataset to obtain semantic feature A; (2) Construct a conditional visual classifier with the image dataset and semantic attribute dataset as inputs, generate pseudo-labels for images of unseen classes, use the obtained pseudo-labels as the labels of unseen classes, and use the unseen class images in the image dataset and the pseudo-labels obtained through the conditional visual classifier to form a new image dataset. The visible class image dataset, unseen class images, and pseudo-labels in the image dataset are used as the training samples of the generalized zero-shot classification model for subsequent classification training; (3) Construct a generalized zero-shot classification model; The generalized zero-shot classification model includes a visual feature refinement module, a semantic feature refinement module, a Swin Transformer network with two added classification tokens, and two constructed self-supervised learning tasks. The two newly added classification tokens correspond to two self-supervised learning tasks: the rotation angle classification task and the contrastive learning task. In the rotation angle classification task, the training samples need to be randomly rotated by four different angles, and finally the rotation category is predicted; The contrastive learning task is to input the training samples into the ResNet101 and Swin Transformer networks respectively to obtain visual features extracted by different feature extractors, and be constrained by the contrastive loss function; Construct a visual feature refinement module: The visual feature refinement module is improved based on the idea of bilinear pooling and is an improved method of feature fusion. It uses the method of homologous bilinear pooling. The input of the visual feature refinement module is the visual feature x obtained after the training samples pass through the Swin Transformer network. The visual feature refinement module makes the visual feature more suitable for fine-grained datasets, including Hadamard operation, reshape operation, fully connected layer, and normalization layer. The specific operation is to copy the visual feature x for subsequent feature fusion. The initial dimension of x is q. After the reshape operation, the dimensions of the two visual features x are changed to 1×q and q×1 respectively, named x1 and x2. x1 is decomposed into the parameter matrix U1 and the feature vector x2 is decomposed into the parameter matrix U2 and the feature vector The product of the parameter matrix U1 and the feature vector is x1, and the product of the feature vector and the parameter matrix U2 is x2. Then, U1 and U2 are input into the global vector layer through the Hadamard operation, and are also input into the global vector layer through the Hadamard operation and aggregated into the global vector z in the global vector layer. After passing through a fully connected layer and a normalization operation, the output is the refined visual feature Construct a semantic feature refinement module: The semantic feature refinement module is called the image adaptive IAS module. The IAS module combines the original semantic features for inter-class discrimination with the image-specific attention vectors for intra-class variations, and then maps the image adaptive semantic features into the corresponding visual space, thereby improving the accuracy of GZSL image classification; The IAS module includes the first fully connected layer FC, the first classification function softmax, the Hadamard operation, the sum operation, the second fully connected layer, the third fully connected layer FC, the first normalization layer, the second normalization layer, and the second classification function softmax. The input of the IAS module is the visual features output by the visual feature refinement module and the semantic feature A obtained by training the semantic attribute dataset using the ResNet101 network. The visual features The result after being processed by the first fully connected layer FC and the first classification function softmax performs the Hadamard operation and the sum operation with the semantic feature to obtain the improved semantic feature The improved semantic feature After being processed by the second fully connected layer FC, the third fully connected layer FC, and the second normalization layer, it performs the Hadamard operation and the second classification function softmax processing with the result of the visual feature x after being processed by the second normalization layer, and maps the semantic feature into the visual space; Input the training samples into the Swin Transformer network. Pass through the visual feature refinement module and the IAS module in sequence to output the classification categories of the generalized zero-shot image classification task, and train the generalized zero-shot classification model. The total loss function of the generalized zero-shot classification model is the sum of the loss function of self-supervised learning and the loss of the generalized zero-shot classification task. The loss function of self-supervised learning includes the loss function of the rotation angle classification task and the loss function of the contrastive learning task, which is expressed by the formula: Among them, is the total loss function of the generalized zero-shot classification model, is the loss function of the rotation angle classification task, is the loss function of the generalized zero-shot classification task, is the loss function of the contrastive learning task; The loss function of the generalized zero-shot classification task is: where M represents the number of training samples, and y i and represent the true label and the predicted label of the generalized zero-shot image classification task, respectively; The loss function of the rotation angle classification task is: Among them, represents the predicted label of the rotation angle classification task, and a ∈ {0, 1, 2, 3} represents 4 rotation angles; The loss function of the contrastive learning task is: where M represents the number of training samples, respectively represent the visual features of the training samples after passing through the ResNet101 network and the SwinTransformer network, W represents the weight matrix of the Swin Transformer network, represents the similarity between; Thus, a trained generalized zero-shot classification model is obtained; (4) Use the trained generalized zero-shot classification model to identify generalized zero-shot images and complete the generalized zero-shot classification task.

2. The generalized zero-shot image classification method according to claim 1, characterized in that, The process of obtaining the pseudo-labels is as follows: The image dataset of visible classes and the semantic attribute dataset in the image dataset are used for training to obtain a visible-class conditional visual classifier conditional on semantic attributes. Then, the image dataset of unseen classes and the semantic attribute dataset are used for training to obtain an unseen-class conditional visual classifier conditional on semantic attributes. The weight matrix W of the visible classes is obtained through the visible-class conditional visual classifier s , and the weight matrix W of the visible classes s is used as the classification weight of the unseen-class conditional visual classifier to obtain the pseudo-labels of the unseen-class image x u ​ 3. The generalized zero-shot image classification method according to claim 1, wherein In step (1), obtaining the image dataset and semantic attribute dataset of the generalized zero-shot classification model specifically is: (1.1) Use existing datasets for image classification and fine-grained image classification as the image dataset, including: Animal with Attribute 2 (AWA2), Caltech-UCSD-Birds-200-2011 (CUB), and SUN Attribute Dataset (SUN). Among them, the CUB and SUN datasets are fine-grained datasets, and the AWA2 is a coarse-grained dataset. The dataset AWA2 contains 30,745 and 37,322 animal pictures of 50 classes, among which 40 classes are used as training categories and 10 classes are used as test categories. The dataset CUB is a fine-grained dataset of bird pictures, containing 11,788 pictures of 200 bird species. Among them, 150 classes are used as training categories and 50 classes are used as test categories. The dataset SUN covers a fine-grained dataset of various environmental scenes and internal images, containing 14,340 pictures of 717 classes. Among them, 645 classes are used as training categories and 72 classes are used as test categories. To meet the input requirements of the backbone network, that is, the Swin Transformer network, the image resolution is unified to 224×224, and the visual features X of the visible class images s and its label Y s are denoted as D s ={X s , Y s}, as the image dataset of the visible class. The visual features X of the unseen class images u and its label Y u are denoted as D u ={X u , Y u}, as the image dataset of the unseen class; (1.2) Zero-shot classification constructs a semantic space with the help of auxiliary semantic information and establishes the interaction relationship between the visual modality and the semantic modality. Among them, the auxiliary semantic information is divided into two categories: auxiliary information based on artificial definition and auxiliary information based on learning. Among them, the auxiliary information based on artificial definition obtains semantic features of different dimensions through the ResNet101 network. The semantic features A obtained by the semantic attribute dataset through the ResNet101 network are expressed as A = A s ∪A u , where the subscript s represents the visible class and u represents the unseen class. Among them, the AWA2 dataset uses 85-dimensional semantic features, the CUB dataset uses 312-dimensional semantic features, and the SUN dataset uses 102-dimensional semantic features.

4. The generalized zero-shot image classification method according to claim 1, characterized in that The four different angles are 0°, 90°, 180°, and 270° respectively, and the rotation labels are set to 0, 1, 2, and 3 respectively.

5. The generalized zero-shot image classification method according to claim 1, characterized in that, In step (3), constructing the generalized zero-shot classification model specifically is: (3.1) Swin Transformer Network with Two Classification Tokens: Using the Swin Transformer network as the backbone network, after the images in the training samples are input into the Swin Transformer network, they become a series of flattened 2D patches x p , where the resolution of each patch is p×p, and the dimension of each patch is set to 48; The flattened patch blocks obtain the visual feature x of the input image after 4 stages. Each stage includes a patch merging layer and a Swin Transformer block. The Swin Transformer block includes a window multi-head self-attention module and a shifted window multi-head self-attention module; (3.2) Two self-supervised task mechanisms: Rotation Angle Classification Task: To construct a task for predicting the rotation angle of an image to achieve self-supervised learning. Specifically, the images in the training samples are randomly rotated by four different angles: 0°, 90°, 180°, and 270°. The training samples are further processed to obtain a rotated image dataset. Four categories are set for the four rotation angles, and the labels are set as 0, 1, 2, and 3. The rotated image dataset is also input into the Swin Transformer network to obtain the visual features with rotation information corresponding to the rotated image dataset, which are used to predict their rotation categories; Contrastive Learning Task: The training samples are respectively input into the ResNet101 network and the Swin Transformer network to obtain the visual features of the same image after different feature extractors. These two visual features processed by different feature extractors are used as positive sample pairs. The similarity between the positive sample pairs is calculated through cosine similarity, and the distance between the positive sample pairs is constrained to be close enough through contrastive loss; (3.3) The input of the visual feature refinement module is the visual feature x processed in step (3.1), and the output is the refined visual feature (3.4) The input of the IAS module is the refined visual features processed in step (3.3) and the semantic feature A processed by the ResNet101 network, and the output is the mapping of the visual feature x and the semantic feature A in the visual space; (3.5) Input the training samples and the rotated image dataset into the Swin Transformer network for feature extraction simultaneously, and respectively obtain their corresponding visual features x. Use the obtained visual features to perform the generalized zero-shot image classification task and the rotation angle classification task respectively. Steps (3.1) and (3.2) are jointly trained and share the same Swin Transformer network; Training a generalized zero-shot classification model, the total loss function of the generalized zero-shot classification model is the sum of the loss function for self-supervised learning and the loss for the generalized zero-shot classification task; The Swin Transformer network is trained together with the visual feature refinement module and the IAS module. The initial iteration number is k, the maximum iteration number is K, K ≥ 30, and let k = 1; Obtain the trained generalized zero-shot classification model.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it can implement the steps of the generalized zero-shot image classification method described in any one of claims 1-5.

Images