A generalized zero-shot image classification method and system for domain correction
By combining generative-adversarial mechanisms and visual-semantic triplet loss, along with generative and classification branch networks, the problem of insufficient labeled samples in deep learning models is solved, improving the accuracy and feature discriminability of generalized zero-shot image classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2024-03-28
- Publication Date
- 2026-07-21
AI Technical Summary
Existing deep learning models require a large number of labeled samples for training, and it is difficult to obtain a large number of samples of all categories in real-world scenarios. Traditional zero-shot learning techniques cannot handle the problem of recognizing both seen and unseen classes simultaneously. The inconsistent technical results are caused by the problem of visual features synthesized by the generator, and there are technical problems with unseen classes in existing technologies.
A generative-adversarial network structure is adopted, which combines a generative branch network and a classification branch network. The generator and classifier are optimized by jointly mining the loss through a generative correction module and a visual-semantic triplet. The generative correction module includes an image feature extraction network, a semantic feature extraction network, a generator, and a discriminator. ResNet-101 and RNN are used for feature extraction and generative correction.
It effectively solves the problem of insufficient label samples or missing label samples in certain categories, improves classification performance, ensures that the distribution of synthetic visual features is consistent with the distribution of real data, enhances the discriminability of features, and improves the accuracy of generalized zero-shot image classification.
Smart Images
Figure CN118262162B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image classification, and in particular to a generalized zero-shot image classification method and system for generative domain correction. Background Technology
[0002] In recent years, with the latest advancements in image processing and computer vision, deep learning models have gained widespread popularity due to their ability to provide end-to-end models from feature extraction to classification. While traditional deep learning models have been successful, they require not only a large number of samples but also extensive labeled data for each class during training. Collecting large-scale labeled samples is a challenging problem. Furthermore, obtaining a large number of samples across all categories is difficult to achieve in real-world scenarios. First, fine-grained annotation of a large number of samples is laborious and requires relevant domain knowledge. Second, many categories lack sufficient labeled samples, such as endangered animals, or data not included during training but appearing in the testing phase. Additionally, standard deep learning models can only identify samples belonging to classes already seen during training, and they cannot handle samples from unseen classes.
[0003] To address the aforementioned problems, zero-shot learning has been proposed. The goal of zero-shot learning is to train a model that classifies unseen classes (target domains) by transferring knowledge from other seen classes (source domains) through semantic information transfer. However, in traditional zero-shot learning techniques, the test set only contains samples from unseen classes, which is an unrealistic setting that fails to reflect real-world recognition conditions. It is necessary to recognize samples from both seen and unseen classes simultaneously, rather than classifying only unseen class data. This learning setting is called generalized zero-shot learning, where the test set includes both seen and unseen classes. Due to the special setting of generalized zero-shot learning, visual features of unseen classes are missing during the training phase. Some works employ generative methods, learning a generative model that generates visual features of unseen classes based on samples from seen classes and the semantic representations of both seen and unseen classes. By generating samples for unseen classes, the generalized zero-shot problem can be transformed into a traditional supervised learning problem.
[0004] However, the above scenario is merely an ideal situation. These generative methods typically employ VAEs or GANs as generators. But due to the use of element-wise distance metrics or unstable training processes, generators may learn noisy or unnatural features, constructing unconventional data distributions rather than broader features that more closely resemble the original data distribution. The aforementioned problems indicate that when we use generators to synthesize visual features for unseen classes to supplement their visual information and train a classifier, it may lead to the classifier learning inaccurate category features. This process can degrade the overall classification performance of the model. Because current models do not consider the inconsistency in the distribution of synthesized features caused by the generator, these models exhibit significant performance degradation when predicting unseen classes. Summary of the Invention
[0005] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide a generalized zero-shot image classification method and system for generative domain correction.
[0006] The objective of this invention is achieved through the following technical solution:
[0007] A generalized zero-shot image classification method for generative domain correction includes:
[0008] Establish training and testing datasets. The training dataset includes images of seen classes, semantic information of attributes of seen classes, and semantic information of attributes of unseen classes. The testing dataset includes images of both seen and unseen classes. For seen classes, the images in the training dataset are unique to each class; for unseen classes, the training dataset does not contain any images.
[0009] Further details are as follows: Two datasets D train D test .
[0010] Training dataset D train Including training data of known classes (Used for training the generator) and training data for unseen classes (used to generate synthetic unseen class visual features), X represents the images used for training, and y represents the entire image set. seen Represents a known category label, y unseen Y represents an unseen category label. seen and Y unseen These represent the sets of tags for seen and unseen classes, respectively, and guarantee that... a seen Represents a known class attribute semantics, a unseenA represents a type of unseen class attribute semantics, and A represents the overall set of attribute semantics.
[0011] During testing, there is no distinction between seen and unseen classes; the test dataset... y represents a class label, Y represents the entire label space, and the above test data is used for classifier performance evaluation. Furthermore, the test dataset D... test Including test data of already seen classes Test data of unseen classes This represents the images that have been seen and are used for testing. This represents an image used for testing that is not currently in use, and guarantees that...
[0012] Construct generative branch network models and classification branch network models;
[0013] The generative branch network model is a network structure based on the generative adversarial mechanism, including an image feature extraction network, a semantic feature extraction network, a generator, a generative correction module, and a discriminator.
[0014] The classification branch network model is a generalized zero-shot classification network, which includes an image feature extraction network, a pre-trained generator in the generation branch network, and a classifier.
[0015] In this method, ResNet-101 convolutional neural network is used as the image feature extraction network, and RNN is used as the semantic feature extraction network.
[0016] Train the generative branch network model and the classification branch network model.
[0017] The generative branch network model includes an image feature extraction network, a semantic feature extraction network, a generator, a generation correction module, and a discriminator. The steps for training the generative branch network model are as follows:
[0018] x images of the known class in the training data seen and attribute semantics a seen The semantics of attributes of unseen classes (a) unseen The input is used to generate a branching network model, and the image feature extraction network is used to obtain the visual features f of the seen classes. seen The semantic feature extraction network is used to obtain the semantic features s of the seen class and the unseen class, respectively. seen and s unseen ;
[0019] The obtained semantic features of the seen class are input into the generator to obtain the synthetic visual features of the seen class. Reconstruct features to optimize the generator; the loss function is:
[0020]
[0021] in, G stands for generator.
[0022] The obtained synthetic visual features of the seen class are input into the generation and correction module to obtain the synthetic corrected visual features of the seen class. Reconstructing features to optimize the generation of the correction module, the loss function is:
[0023]
[0024] in, H represents the generation and correction module.
[0025] The visual and semantic features of negative examples from the observed class are obtained by random sampling. The generated components are then optimized using a visual-semantic triplet mining loss function. The loss function is as follows:
[0026]
[0027] The marginal value Δ is fixed at 20. To synthesize corrected visual features for existing classes, i.e. These are existing, real visual features of the same category, used as positive example visual feature samples. These are real visual features of previously seen classes that are not of the same category, and are used as negative example visual feature samples. All of them are existing semantic features of the same category. Seen semantic features that are not of the same category are used as negative example semantic feature samples.
[0028] The obtained corrected visual features and real visual features of the seen classes are input into the discriminator to obtain the predicted values corresponding to the features. The branch network is then optimized using the predicted values, and the loss function is:
[0029]
[0030] Where D represents the discriminator and the interpolation feature. The penalty coefficient λ is fixed at 10.
[0031] The total losses are as follows:
[0032] L all =L rec +L adv +αL cal +βL tj
[0033] Here, α and β are both loss weighting factors, which are fixed at 0.05 and 0.3, respectively.
[0034] The classification branch network model includes an image feature extraction network, a pre-trained generator in the generation branch network, and a classifier. The steps for training the classification branch network model are as follows:
[0035] The unseen class semantic features s obtained from the aforementioned generative branch network unseen Input the pre-trained generator into the generative branch network to obtain synthetic visual features of unseen classes.
[0036] The training dataset contains real-world visual features f. seen Synthetic visual features of unseen classes Input the classifier, output the category prediction result;
[0037] The classification network model is optimized using the prediction results and the cross-entropy function.
[0038] The losses are as follows:
[0039]
[0040] Where k∈[1,K] represents the category label, K is the total number of categories, and I [k＝y] It is an indicator function that indicates that the function is 1 when the actual class label y is the same as the class label k, and 0 otherwise. x is the visual feature input to the classifier.
[0041] Furthermore, the data signal can be simply described as follows:
[0042] During the training phase, the seen class images x in the input training data are used to generate the branch network model. seen Seen class attribute semantics a seen No class attribute semantics found. unseen Output: Visual features f (as seen) seen , Seen synthetic visual features Seen synthetic corrected visual features and unseen semantic features s unseen Used to calculate reconstruction loss L rec , Generate correction loss L cal Visual-semantic triplet joint mining loss L tj Adversarial generation loss L adv Optimize the generated branch network model.
[0043] The classification network model first inputs the unseen class semantic features s obtained through the generative branch network. unseen The generator in the generative branch network is fed into a pre-trained generator, which outputs unseen class synthetic visual features. Further input is the visual features f of the observed classes obtained through the generative branching network. seen No synthetic visual features were observed. The classifier is fed the actual category label y from the training data and outputs the category prediction probability p(y|x), which represents the probability of the category corresponding to the visual feature.
[0044] The trained classification branch network model is used to predict the category of the test data, specifically as follows:
[0045] First, an image feature extraction network is used to extract image x from the test dataset. test visual features f test ;
[0046] Then, a softmax-based classifier can be used to classify each feature sample.
[0047] The specific classification is as follows:
[0048]
[0049] Where y′ represents the predicted class label, p(f) represents the predicted probability result of the input visual feature, which can be further expanded into the relationship between the weight of the softmax classifier and the visual feature, n∈[0,N] represents the dimension of a visual feature, and N is the total dimension of the visual feature.
[0050] A system for a generalized zero-shot image classification method includes:
[0051] Dataset module: Obtain the training dataset and test dataset;
[0052] Modules: Construct generative branch network models and classification branch network models;
[0053] Training module: Trains the generative branch network model and the classification branch network model;
[0054] Prediction module: Uses a trained classification branch network model to predict the category of the test data.
[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0056] (1) Traditional image classification models require a large number of balanced labeled samples to achieve satisfactory results. The method proposed in this paper can solve the problem of insufficient labeled samples or missing labeled samples of certain categories. It can achieve good classification performance without using visual features of certain categories.
[0057] (2) Existing generative generalized zero-shot classification methods do not consider the situation where the feature distribution synthesized by the generator is inconsistent with the distribution of real data, resulting in a significant drop in classification performance in such cases. By using generator domain correction, noise or specific features captured by the generator can be effectively removed, making the synthetic visual feature distribution closer to the real visual feature distribution.
[0058] (3) Generative generalized zero-shot methods for fine-grained classification suffer from unclear boundaries between synthesized feature categories. We propose a visual-semantic triplet joint mining loss. By incorporating semantic modality relational constraints, we further reduce the intra-class distance of synthesized visual features and increase their inter-class distance, thereby achieving good feature discriminative ability. Attached Figure Description
[0059] Figure 1 This is a flowchart of the process of this invention;
[0060] Figures 2(a), 2(b), and 2(c) are data example diagrams of embodiments of the present invention;
[0061] Figure 3 This is a schematic diagram of the network structure of the present invention;
[0062] Figure 4(a) is a schematic diagram of the nested residual dense block structure used in this invention;
[0063] Figure 4(b) is a schematic diagram of the residual dense block structure used in this invention. Detailed Implementation
[0064] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.
[0065] like Figure 1 As shown, a single-sample image analysis method with independent training datasets includes the following steps:
[0066] S1 establishes a training dataset and a test dataset. The training dataset includes images of seen classes, semantic information of the attributes of seen classes, and semantic information of the attributes of unseen classes. The test dataset includes images of both seen and unseen classes. For seen classes, the images in the training dataset are not duplicated with the images in the test dataset; for unseen classes, the training dataset does not contain any images.
[0067] In this embodiment, three datasets are used as model testing datasets: Caltech-UCSD Birds-200-2011 (CUB), Animals with Attributes 2 (AWA2), and SUN attributes (SUN). The CUB dataset contains 11,788 images, 200 bird category labels, and 312 attributes. The AWA2 dataset contains 37,322 images, 50 animal category labels, and 85 attributes. The SUN dataset contains 14,340 images, 717 scene category labels, and 102 attributes. Figures 2(a), 2(b), and 2(c) are from the CUB, AWA2, and SUN datasets, respectively.
[0068] S2 constructs generative branch network models and classification branch network models;
[0069] The image feature extraction network uses a ResNet-101 convolutional neural network. ResNet-101 employs a hierarchical structure, dividing the entire network into five layers. Specifically, the 2048-dimensional features encoded after the fifth convolutional group are used as the output feature map of the image feature extraction network. The overall network structure is as follows: Figure 3 The generation correction module, which implements generation domain correction, consists of four nested residual dense blocks. Each nested residual dense block is formed by connecting three residual dense blocks through residuals. The structure of the nested residual dense blocks is shown in Figure 4(a), and the structure of the residual dense block is shown in Figure 4(b).
[0070] The generative branch network model includes an image feature extraction network, a semantic feature extraction network, a generator, a generative correction module, and a discriminator.
[0071] The classification branch network model includes an image feature extraction network, a pre-trained generator in the generation branch network, and a softmax-based classifier.
[0072] S3 training generates branch network models and classification branch network models;
[0073] During the training phase, the feature map output by the image feature extraction network has a size of 2048×1. Under the generalized zero-shot learning setting, the CUB dataset contains 150 seen classes and 50 unseen classes, the AWA2 dataset contains 40 seen classes and 10 unseen classes, and the SUN dataset contains 645 seen classes and 72 unseen classes. The generative branch network was trained using the Adam optimizer for 48,300 iterations, with the generator, discriminator, and generative correction modules all having a learning rate of 1e-4 and a batch size of 64. The classification branch network was also trained using the Adam optimizer, with the classifier having a learning rate of 1e-3.
[0074] S4 uses a trained classification branch network model to predict the categories of the test dataset.
[0075] During the testing process, the final result can be obtained using only the classification branch network. Specifically, the image to be tested is input into the classification branch network, which first extracts features through the image feature extraction network, and then uses a softmax-based classifier to classify each feature sample.
[0076] This embodiment also provides a generalized zero-shot image classification system, including:
[0077] Dataset module: Obtain the training dataset and test dataset;
[0078] Modules: Construct generative branch network models and classification branch network models;
[0079] Training module: Trains the generative branch network model and the classification branch network model;
[0080] Prediction module: Uses a trained classification branch network model to predict the category of the test dataset.
[0081] This method utilizes a generative correction module and a well-designed generative-adversarial network structure to alleviate the performance degradation problem caused by the inconsistency between the feature distribution synthesized by the generator and the real data distribution in generative generalized zero-shot classification methods. Simultaneously, a visual-semantic triplet state joint mining loss is proposed to further ensure that the intra-class distance of synthesized visual features is reduced while the inter-class distance is increased, thereby enhancing the discriminative power of synthesized visual features. Deep networks trained using this method can alleviate the inconsistency in the synthesized feature distribution of generative generalized zero-shot classification methods and further improve classification performance.
[0082] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A generalized zero-shot image classification method for generative domain correction, characterized in that, include: Establish a training dataset and a test dataset. The training dataset includes images of seen classes, attribute semantic information of seen classes, and attribute semantic information of unseen classes. The test dataset includes images of seen classes and images of unseen classes. For seen classes, the images in the training dataset and the images in the test dataset are not duplicated. For unseen classes, the training dataset does not contain any images. Construct generative branch network models and classification branch network models; The generative branch network model is a network structure based on a generative-adversarial mechanism; The classification branch network model is a generalized zero-shot image classification network based on generated images. Train generative branch network models and classification branch network models; The trained classification branch network model is used to predict the category of the test dataset; The generative branch network model includes an image feature extraction network, a semantic feature extraction network, a generator, a generation correction module, and a discriminator. The steps for training the generative branch network model are as follows: Images that have already been classified in the training data and attribute semantics Attribute semantics of unseen classes The input is used to generate a branching network model, and the image feature extraction network is used to obtain the visual features of the seen classes. The semantic feature extraction network is used to obtain the semantic features of seen and unseen classes respectively. and ; The obtained semantic features of the seen class are input into the generator to obtain the synthetic visual features of the seen class. Reconstruct features to optimize the generator; the loss function is: ； in, , Represents a generator; The obtained synthetic visual features of the seen class are input into the generation and correction module to obtain the synthetic corrected visual features of the seen class. Features are reconstructed to optimize the generation of the correction module, and the loss function is: ； in, , This represents the generation and correction module; The visual and semantic features of negative examples of the seen class are obtained by random sampling. The generated components are optimized by using a visual-semantic triplet joint mining loss function, which is: ； Among them, marginal value Fixed at 20, To synthesize corrected visual features for existing classes, i.e. , These are existing real visual features of the same category, used as positive example visual feature samples. These are real visual features of previously seen classes that are not of the same category, and are used as negative example visual feature samples. All of them are existing semantic features of the same category. Seen semantic features that are not of the same category are used as negative example semantic feature samples; The obtained corrected visual features and real visual features of the seen classes are input into the discriminator to obtain the predicted values corresponding to the features. The branch network is then optimized using the predicted values, and the loss function is: ； in, Discriminator, interpolation features , Penalty coefficient Fixed at 10; The total losses are as follows: in, and All of these are loss weighting factors.
2. The generalized zero-shot image classification method for generative domain correction according to claim 1, characterized in that, The classification branch network model includes an image feature extraction network, a pre-trained generator and classifier for the generation branch network.
3. The generalized zero-shot image classification method for generative domain correction according to claim 2, characterized in that, The classifier is specifically a softmax-based classifier.
4. The generalized zero-shot image classification method for generative domain correction according to claim 3, characterized in that, The training of the classification branch network model is specifically as follows: The unseen class semantic features obtained from the generative branch network are input into the generator that has been trained in the generative branch network to obtain the synthetic visual features of the unseen class. Input the real visual features of seen classes and the synthetic visual features of unseen classes from the training data into the classifier, and output the class prediction results. The classification network model is optimized using the prediction results and the cross-entropy function.
5. The generalized zero-shot image classification method for generative domain correction according to claim 4, characterized in that, The loss function of the classification branch network is: in, Represents category label, The total number of categories, It is an indicator function that represents the actual category label. With category label The function evaluates to 1 if the values are the same, and 0 otherwise. The visual features are input to the classifier. Predict the probability for the output category.
6. The generalized zero-shot image classification method for generative domain correction according to any one of claims 1-5, characterized in that, The process of using a trained classification branch network model to predict the category of the test data specifically involves: First, an image feature extraction network is used to extract images from the test dataset. visual features ; Then, a softmax-based classifier can be used to classify each feature sample.
7. A system based on the generalized zero-shot image classification method for generative domain correction as described in any one of claims 1-6, characterized in that, include: Dataset module: Obtain the training dataset and test dataset; Modules: Construct generative branch network models and classification branch network models; Training module: Trains the generative branch network model and the classification branch network model; Prediction module: Uses a trained classification branch network model to predict the category of the test dataset.