A Zero-Shot Classification Method Based on Data Augmentation

Abstract

This invention discloses a zero-shot classification method based on data augmentation. The method includes the following steps: First, visual features are combined, and the resulting combined visual features are input into a relational network. The output score is applied within the visible class range, achieving data augmentation for the visible class. Then, visual features are combined with semantic features of the unseen class, and the resulting combined features are input into the relational network. The output score is applied across all classes, achieving data augmentation for the unseen class. Finally, by integrating triples obtained from matching visual features, randomly shuffled visual features, and semantic features of the unseen class, the ability of the zero-shot model to distinguish between the unseen and visible classes is improved through data augmentation. This method improves the accuracy of zero-shot image classification methods through three data augmentation techniques applicable to zero-shot classification.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a zero-shot classification method based on data augmentation. Background Technology

[0002] The emergence of numerous publicly available datasets is one of the reasons for the rapid development of the computer vision field. However, this reliance on large amounts of labeled data is also hindering the widespread application of deep learning models in real-world scenarios. Most computer vision models require the support of large, high-quality labeled datasets to perform well on their respective tasks. However, collecting, cleaning, labeling, and proofreading such datasets requires a significant investment of time and resources, making it difficult for individuals and small businesses to create such high-quality datasets for a single problem. Furthermore, tasks requiring even more refined labeling, such as object detection and semantic segmentation, will consume even more time and resources in collecting training datasets for these tasks.

[0003] To reduce the dependence of computer vision models on data, researchers have proposed numerous methods. On one hand, they utilize web crawlers and other methods to acquire vast amounts of unlabeled, weakly labeled, and imprecisely labeled data readily available on the internet. By adding non-human-labeled data, they expand the training dataset, helping the model improve its capabilities. On the other hand, they propose learning methods such as few-shot learning and zero-shot learning, achieving the same experimental results as large datasets using only a small amount of labeled training data. Zero-shot learning, as an extreme case of few-shot learning, excludes unseen class labeled data during the training phase. This means that ordinary deep learning models only encounter labeled data of visible classes during training and cannot recognize unseen test samples during testing. Addressing the zero-shot problem requires models that differ from traditional deep learning methods.

[0004] Currently, zero-shot learning is mainly divided into two categories: embedded methods and generative methods. Embedded methods embed visual and semantic features into the same space. During the training phase, they learn the mapping relationship between visible class visual features and semantic features. In the usage phase, they transfer the learned mapping relationship to the recognition of unseen classes, thus achieving zero-shot image classification. Generative methods, on the other hand, transform the zero-shot problem into a regular fully supervised problem by artificially generating visual features for unseen classes, thereby addressing the issue that the model has not encountered unseen classes during training. Both embedded and generative methods have their advantages and disadvantages. This method optimizes and improves the embedded method to address its shortcomings. The advantages of embedded methods are fast training speed, ease of increasing the number of unseen classes, and the ability to achieve end-to-end prediction. Its disadvantage is that it cannot utilize the semantic information of unseen classes, resulting in prediction results biased towards visible classes, leading to lower prediction accuracy compared to generative methods. Summary of the Invention

[0005] (1) Technical issues to be resolved

[0006] To address the shortcomings of current embedded methods, such as the inability to utilize semantic information of unseen classes and the resulting bias towards seen classes, leading to lower accuracy compared to generative methods, a zero-shot classification method based on data augmentation is proposed.

[0007] (2) Technical solution

[0008] The objective of this invention is achieved through the following technical solution: a zero-shot classification method based on data augmentation, comprising the following steps:

[0009] Step 1: Use a convolutional neural network to extract visual features from the training images, and use a recurrent neural network to extract semantic features from the category names of all categories.

[0010] Step 2: Map the visual and semantic features extracted in Step 1 to a space of the same dimension using different multilayer perceptrons to ensure that the feature dimensions of the visual and semantic features are the same after mapping.

[0011] Step 3: After copying the visual features of a training image sample obtained in Step 2 to the number of visible class categories, each copy is concatenated with the semantic features of the visible class obtained in Step 2. The concatenated features are then input as a whole into the RelationNet for class confidence prediction. The loss is calculated based on the labeled class of the training sample.

[0012] Step 4: Randomly combine the mapped visual features obtained from Step 2 of the same batch of training. After copying the combined features to the number of visible class categories, concatenate each copy with the mapped semantic features of a visible class. Input the concatenated features as a whole into the relational network for class confidence prediction. Calculate the loss 2 based on the combined class label of the combined features.

[0013] Step 5: Randomly combine the mapping visual features obtained in Step 2 with the mapping category semantic features of the unseen classes to obtain the combined visual and semantic features. After copying the combined features to the number of categories of all classes, each copy is concatenated with the mapping semantic features of one class. The concatenated features are input as a whole into the relational network for category confidence prediction. The loss 3 is calculated based on the combined category label of the combined features.

[0014] Step 6: Concatenate the mapping visual features of the same training batch with the corresponding mapping semantic features, input the concatenated features into the relational network to output the visibility class score, copy the result, and randomly shuffle the copy to obtain a random visibility class score.

[0015] Step 7: After concatenating the mapping visual features of the same training batch with the mapping semantic features of the unseen class, input the data into the relational network to obtain the score of the unseen class.

[0016] Step 8: Calculate the triplet loss based on the three scores obtained in Steps 6 and 7: the visible class score, the randomly visible class score, and the unseen class score.

[0017] Step 1: Add the losses 1, 2, 3 and triplet losses from steps 3, 4, 5 and 8 to obtain the total loss of the model during the training phase.

[0018] Step 10: Use the total loss obtained in Step 9 to train the relational network in Step 3 and the multilayer perceptron in Step 2;

[0019] Step 11: In the prediction phase, the image to be tested is concatenated with the semantic features of all categories and input into the trained relational network. The score of each category is output, and the category with the highest score is taken as the predicted category of the image to be tested.

[0020] As a preferred technical solution, in step one, the visual features of the image have a dimension of 2048, and the category semantic features extracted by the recurrent convolutional neural network from the category name have a dimension of 1024.

[0021] As a preferred technical solution, in step two, two four-layer perceptrons, each consisting of a fully connected layer, a ReLU activation function, a fully connected layer, and a LeakyReLU activation function, are used to map the 2048-dimensional visual features and the 1024-dimensional semantic features to the 2048-dimensional same-dimensional space.

[0022] As a preferred technical solution, the relationNet in step three consists of four layers: a fully connected layer, a LeakyReLU activation function, a fully connected layer, and a Sigmoid activation function. The input of the relationNet is 4056-dimensional features, and the output is a score between 0 and 1.

[0023] As a preferred technical solution, in step three, the formula for calculating loss 1 is as follows:

[0024]

[0025] Where N and S represent the number of samples and the number of visible classes in a training batch, respectively; y is the class indicator number, which equals 1 when the labeled class of sample i is the same as that of j, otherwise y equals 0; v represents the output of the concatenated features in step three of the relational network input.

[0026] As a preferred technical solution, the formula for calculating the combined category label of visual features and combined features randomly combined from the same training batch in step four is as follows:

[0027]

[0028]

[0029] Where m and n represent two different sample numbers, and x and y represent visual features and category labels, respectively.

[0030] As a preferred technical solution, the formula for calculating loss 2 in step four is as follows:

[0031]

[0032] As a preferred technical solution, the calculation formula for the combined category label of the combined visual features, the unseen semantic features, and the combined features in step five is as follows:

[0033]

[0034]

[0035] Where 'a' represents the semantic features of unseen classes.

[0036] As a preferred technical solution, the formula for calculating loss 3 in step five is as follows:

[0037]

[0038] As a preferred technical solution, the triplet loss formula in step eight is as follows:

[0039] L tri =max(||x a -x p ||-||x a -x n ||+α,0);

[0040] Where x a x p and x n These are the scores for the visible class, the randomly visible class, and the unseen class, respectively.

[0041] (3) Beneficial effects

[0042] The beneficial effects of this invention are as follows: This method helps the embedded zero-shot method overcome its shortcomings through data augmentation. Specifically, this method combines visual features to obtain visual composite features, inputs them into a relational network, and classifies the output results within the visible class range, achieving the purpose of visible class data augmentation; it combines visual features with unseen class semantic features, inputs them into a relational network, and classifies the output results within the entire class range, achieving the purpose of unseen class data augmentation; and it increases the zero-shot model's ability to distinguish between unseen and visible classes through triples obtained by matching visual features, randomly shuffled visual features, and unseen class semantic features, thereby augmenting the data. Attached Figure Description

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 It is the overall model structure;

[0045] Figure 2 This is a schematic diagram of a four-layer multilayer sensor;

[0046] Figure 3 This is a diagram of a relationship network. Detailed Implementation

[0047] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown, the objective of this invention is achieved through the following technical solution: The main steps are as follows:

[0049] Step 1: Use a convolutional neural network to extract 2048-dimensional visual features from the training images, and use a recurrent neural network to extract 1024-dimensional semantic features from the category names of all categories.

[0050] Step 2: The visual and semantic features extracted in Step 1 are respectively passed through two four-layer multilayer perceptrons consisting of a fully connected layer, a ReLU activation function, a fully connected layer, and a Leaky ReLU activation function, to a 2048-dimensional space. The structure of the four-layer multilayer perceptron is as follows: Figure 2 As shown.

[0051] Step 3: After copying the mapped visual features of a training image sample obtained in Step 2 to the number of visible class categories, concatenate each copy with the mapped semantic features of the visible class obtained in Step 2. Input the concatenated combined features as a whole into a relationNet network composed of fully connected layers, LeakyReLU activation function, fully connected layers, and a Sigmoid activation function for class confidence prediction. Calculate the loss 1 based on the labeled class of the training sample. The relationNet structure is as follows: Figure 3 As shown.

[0052] Step 4: Randomly mix the mapped visual features obtained from Step 2 in the same batch of training. After copying the resulting mixed features to the number of visible class categories, concatenate each copy with the semantic features of a visible class. Input the concatenated combined features as a whole into the relational network for class confidence prediction. Calculate the loss 2 based on the mixed class label of the mixed samples.

[0053] Step 5: Randomly combine the mapped visual features obtained in Step 2 with the mapped semantic features of the unseen classes obtained in Step 2 to obtain combined visual and semantic features. After copying the mixed features to the number of classes, each copy is concatenated with the mapped semantic features of a class. The concatenated combined features are input as a whole into the relational network for class confidence prediction. The loss is calculated based on the class labels obtained from the combination of semantic and visual features.

[0054] Step 6: After concatenating the mapped visual features of the same training batch with the corresponding mapped semantic features, input the result into the relational network to output the visibility class score. Copy the result and randomly shuffle the copy to obtain a random visibility class score.

[0055] Step 7: After concatenating the mapped visual features of the same training batch with the mapped semantic features of the unseen class, input the concatenated features into the relational network to obtain the scoring results of the unseen class.

[0056] Step 8: Calculate the triplet loss based on the three scores obtained in Steps 5 and 6.

[0057] Step 9: Add the losses 1, 2, 3 and triplet losses from Steps 3, 4, 5 and 8 to obtain the total loss of the model during the training phase.

[0058] Step 10: Use the total loss obtained in Step 9 to train the relational network mentioned in Step 3 and the fully connected layer mentioned in Step 2.

[0059] Step 11: In the prediction phase, the image to be tested is concatenated with the semantic features of all categories and input into the converged relational network. The score of each category is output, and the category with the highest score is taken as the predicted category of the image to be tested.

[0060] It should be noted that this invention achieves data augmentation by constructing an embedded zero-shot model and utilizing combined data obtained from combining visual and semantic features. Simultaneously, it reduces the probability of model classification errors by constructing triples. The key lies in how to obtain the combined data and how to construct triples to achieve zero-shot embedded data augmentation. First, combined data obtained by combining visual features achieves data augmentation for the visible class. Second, combined visual features and semantic features of the unseen class achieve data augmentation for the unseen class. Finally, triples obtained by randomly shuffling visual features and unseen class features enhance the model's ability to distinguish between the visible and unseen classes through data augmentation.

[0061] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the concept and scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the design concept of the present invention should fall within the protection scope of the present invention. The technical content for which protection is sought in the present invention has been fully described in the claims.

Claims

1. A zero-shot classification method based on data augmentation, characterized in that, Includes the following steps: Step 1: Use a convolutional neural network to extract visual features from the training images, and use a recurrent neural network to extract semantic features from the category names of all categories. Step 2: Map the visual and semantic features extracted in Step 1 to a space of the same dimension using different multilayer perceptrons to ensure that the feature dimensions of the visual and semantic features are the same after mapping. Step 3: After copying the visual features of a training image sample obtained in Step 2 to the number of visible class categories, each copy is concatenated with the semantic features of the visible class obtained in Step 2. The concatenated features are then input as a whole into the RelationNet for class confidence prediction. The loss is calculated based on the labeled class of the training sample. Step 4: Randomly combine the mapped visual features obtained from Step 2 of the same batch of training. After copying the combined features to the number of visible class categories, concatenate each copy with the mapped semantic features of a visible class. Input the concatenated features as a whole into the relational network for class confidence prediction. Calculate the loss 2 based on the combined class label of the combined features. Step 5: Randomly combine the mapping visual features obtained in Step 2 with the mapping category semantic features of the unseen classes to obtain the combined visual and semantic features. After copying the combined features to the number of categories of all classes, each copy is concatenated with the mapping semantic features of one class. The concatenated features are input as a whole into the relational network for category confidence prediction. The loss 3 is calculated based on the combined category label of the combined features. Step 6: Concatenate the mapping visual features of the same training batch with the corresponding mapping semantic features, input the concatenated features into the relational network to output the visibility class score, copy the result, and randomly shuffle the copy to obtain a random visibility class score. Step 7: After concatenating the mapping visual features of the same training batch with the mapping semantic features of the unseen class, input the data into the relational network to obtain the score of the unseen class. Step 8: Calculate the triplet loss based on the three scores obtained in Steps 6 and 7: the visible class score, the randomly visible class score, and the unseen class score. Step 9: Add the losses 1, 2, 3 and triplet losses from Steps 3, 4, 5 and 8 to obtain the total loss of the model during the training phase; Step 10: Use the total loss obtained in Step 9 to train the relational network in Step 3 and the multilayer perceptron in Step 2; Step 11: In the prediction phase, the image to be tested is concatenated with the semantic features of all categories and input into the trained relational network. The score of each category is output, and the category with the highest score is taken as the predicted category of the image to be tested.

2. The zero-shot classification method based on data augmentation according to claim 1, characterized in that, In step one, the visual features of the image have a dimension of 2048, and the category semantic features extracted by the recurrent convolutional neural network from the category name have a dimension of 1024.

3. The zero-shot classification method based on data augmentation according to claim 1, characterized in that, In step two, two four-layer perceptrons, each consisting of a fully connected layer, a ReLU activation function, a fully connected layer, and a LeakyReLU activation function, are used to map the 2048-dimensional visual features and the 1024-dimensional semantic features to the same 2048-dimensional space.

4. The zero-shot classification method based on data augmentation according to claim 1, characterized in that, The relation network (relationNet) in step three consists of four layers: a fully connected layer, a LeakyReLU activation function, a fully connected layer, and a Sigmoid activation function. The input to the relation network is 4056-dimensional features, and the output is a score between 0 and 1.

5. The zero-shot classification method based on data augmentation according to claim 1, characterized in that, In step three, the formula for calculating loss 1 is as follows: Where N and S represent the number of samples and the number of visible classes in a training batch, respectively; y is the class indicator number, which equals 1 when the labeled class of sample i is the same as that of j, otherwise y equals 0; v represents the output of the concatenated features in step three of the relational network input.

6. The zero-shot classification method based on data augmentation according to claim 1, characterized in that, The formula for calculating the combined category label of visual features and combined features randomly combined from the same training batch in step four is as follows: Where m and n represent two different sample numbers, and x and y represent visual features and category labels, respectively.

7. The zero-shot classification method based on data augmentation according to claim 1, characterized in that, The formula for calculating loss 2 in step four is as follows:

8. The zero-shot classification method based on data augmentation according to claim 1, characterized in that, The formula for calculating the combined category label of the combined visual features, the unseen semantic features, and the combined features in step five is as follows: Where 'a' represents the semantic features of unseen classes.

9. A zero-shot classification method based on data augmentation according to claim 1, characterized in that, The formula for calculating loss 3 in step five is as follows:

10. A zero-shot classification method based on data augmentation according to claim 1, characterized in that, The triplet loss formula in step eight is as follows: L tri =max(||x a -x p ||-||x a -x n ||+α,0); Where x a x p and x n These are the scores for the visible class, the randomly visible class, and the unseen class, respectively.

Images