An image classification method based on zero-shot learning

Abstract

The application discloses an image classification method based on zero sample learning, and the method comprises the following steps: firstly, an image classification system is built, a visible class data set is used as a training set, and an invisible class data set is used as a test set, and visual global and semantic features are obtained through a CNN visual backbone and GloVe. Then, different dimension features are extracted from the visual and semantic features and fused into multi-element high dimension fusion visual features. After that, the fusion visual features are mapped to a semantic space by using a mapping function, semantic embedding is obtained, and the semantic embedding is optimized by an optimization module. After training, the semantic embedding of a test instance is obtained, a nearest neighbor matching and an explicit self-calibration are used to predict a label, a dot product of the semantic embedding of the test sample and a category semantic vector is calculated, and a category with the highest compatibility score is selected as a predicted category. The application can solve the problem that different dimension features are not considered adequately in zero sample classification, ensure the prediction distribution consistency of visible and invisible classes, and improve the generalization ability for invisible classes.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and particularly relates to an image classification method based on zero-sample learning. Background Art

[0002] Zero-shot learning (ZSL), a novel deep learning method, aims to overcome reliance on label annotations and address the challenge of recognizing unseen data. Inspired by the human process of learning new things, zero-shot learning leverages known training data to explore the inherent connections between seen and unseen classes, enabling recognition of unseen classes during the inference phase. Zero-shot learning bridges the gap between seen and unseen classes using semantic information. It extracts semantically corresponding attribute information from seen training data and extends this information to the recognition and classification of unseen classes, enhancing cross-class generalization and enabling accurate recognition of unseen classes. Training requires meticulously designed algorithms to align visual image information with semantic information for seen classes while also considering generalization to unseen classes. During testing, an ideal zero-shot learning model should be able to predict the most relevant semantics for unseen instances. Two key challenges with zero-shot learning are: first, how to correlate information from two different modalities: visual and semantic; and second, overfitting of the visual and semantic information of seen classes during training can lead to biased inference results toward seen classes during testing.

[0003] Early zero-shot learning methods first embedded features of the seen class into a semantic space before classifying the unseen class. Because these models rely on Bayesian probabilistic predictions and overly rely on labeled data, overfitting is common in zero-shot learning. To address this, some models have begun to utilize global features. However, global features alone cannot accurately capture local features that directly correspond to semantic attributes. For example, if the distinguishing feature of a class is eye color, global features reflect overall appearance rather than local details. Directly using global feature embeddings may not accurately reflect this discriminative attribute, resulting in unsatisfactory results.

[0004] Due to the limitations of global visual features, the mapping relationship between visual and semantic features is unclear. Therefore, some methods combine attention mechanisms with models to locate local image attributes, but lack the ability to represent the relationships between local features. Later, methods further used region graph learning to enhance local reasoning of visual features, improving the model's ability to address visual-semantic interactions. However, these models still have the following flaws: First, the integrated attention mechanism cannot effectively locate and represent local features; second, these methods suffer from bias when dealing with both seen and unseen classes. Summary of the Invention

[0005] The purpose of the present invention is to provide an image classification method based on zero-shot learning, which can solve the problem of insufficient consideration of features of different dimensions in zero-shot classification, ensure the consistency of the predicted distribution of visible and invisible classes, and improve the generalization ability of invisible classes.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] An image classification method based on zero-shot learning includes the following steps:

[0008] Step 1: Build an image classification system, which includes a feature extraction module, a feature fusion module FFT, a feature interaction module FIN and an optimization module;

[0009] Step 2: Use the visible class dataset as the training set and the unseen class dataset as the test set, and obtain visual global features and semantic features through the CNN visual backbone and GloVe;

[0010] Step 3: Extract features of different dimensions from visual features and semantic features through the feature extraction module, and fuse them in the feature fusion module FFT to generate a fused visual feature containing multiple high dimensions; the features of different dimensions include global features, local features, mixed features, and significant local features;

[0011] Step 4: In the feature interaction module FIN, the fused visual features are mapped to the semantic space through the mapping function to obtain the semantic features of the corresponding visual features. Then optimize the system through the optimization module;

[0012] Step 5: After training is completed, first obtain the test instance x i Semantic embedding in semantic space Then use nearest neighbor matching and explicit self-calibration to predict x i Tags c * ;

[0013]

[0014] where c * represents the predicted label, C u / C correspond to zero-shot image classification under CZSL / GZSL settings, respectively. c is the semantic embedding of category c, II is the indicator function;

[0015] By calculating the semantic embedding of the test sample and category semantic vector z c The dot product represents the compatibility or similarity score between the sample feature and the category semantics; Finally, for each test sample x i, the system will select the category c with the highest compatibility score as the predicted category.

[0016] Furthermore, the process of obtaining the visual global features and semantic features in step 2 is as follows:

[0017] In generalized zero-shot learning, the training data of the visible class is used Total C seen categories, of which Represents training sample i, Y i s Represents the category corresponding to sample i; there is also a dataset of unseen classes Represents the unseen class training sample i, Y i u Represents the category corresponding to the unseen class training sample i; there is a category c∈C seen ∪C unseen = C, whose semantic vector set is z containing attribute α c ={z1,z2,…,z α};

[0018] According to the attribute description in the semantic vector set, the language model GloVe is used to extract the attribute name f from each attribute. sem The semantic attribute information F is learned from sem ; Use CNN to extract the original features f cv Visual feature extraction global feature F g , the formula is as follows:

[0019] F g ,F sem =CNN(f cv ),GloVe(f sem ).

[0020] Furthermore, the generation of the fused visual features in step 3 includes the following steps:

[0021] (1) Global feature extraction: Use step 2 to extract the global feature F g Methods;

[0022] (2) Combining visual features with semantic features to enhance the expressive power of features and enable classification of unseen categories;

[0023] The interaction between visual features and semantic features is calculated to generate the interaction feature α, which is then normalized to generate the feature weight A, so as to convert the interaction feature into a weight for subsequent feature calculation. The formula is:

[0024]

[0025] where αi is the i-th element of the input interaction feature α in the last dimension, j is the index of the sum of all categories, and the sum is the sum of all elements in the last dimension;

[0026] (3) Local feature extraction

[0027] Use feature weight A to adjust global feature F g Perform weighting to generate new weighted visual features, namely new high-level local features F l , better capture the details of the image, the formula is as follows:

[0028]

[0029] Where r is the index of the summation process, ranging from 1 to R, and R is the size of the last dimension of the feature;

[0030] (4) Hybrid feature extraction

[0031] Calculate the global feature F g With local features F l The feature weight λ between them is then used to adjust the global feature F g With local features F l Perform weighted summation to generate mixed features F mix , mixed feature F mix The formula is as follows:

[0032] F mix =λ·F g +(1-λ)·F l

[0033] (5) Extraction of salient local features

[0034] By calculating the global feature F g and local features F l The element-wise maximum value of max , F max The calculation formula is as follows:

[0035] F max =max(F g ,F l )

[0036] (6) Feature Fusion

[0037] The global feature F g , local features F l , mixed features F mix and the salient local features F max The final feature representation F is obtained by fusion, and the formula is as follows:

[0038]

[0039] in is the fusion coefficient.

[0040] Furthermore, the semantic embedding described in step 4 The acquisition process is: using semantic feature F sem Semantic attribute vector in As a support vector, based on a mapping function M, the local enhanced visual features F are combined with the semantic attribute information V α Matching, mapping the fused visual features F to the semantic attribute information V through the mapping function M α Consistent semantic embedding space, the output of the feature interaction module FIN, that is, the embedding vector obtained by mapping to the semantic attribute space Where W is the embedding matrix that embeds the fusion feature F into the semantic attribute space, is the semantic attribute vector V α The transpose of .

[0041] Furthermore, the optimization module in step 4 optimizes the system by introducing the loss function To achieve this, the loss function is in is the attribute regression loss function, λ AR is the attribute regression loss weight, is the attribute cross entropy loss function, is the debiasing loss function, λ DEB is the debiasing loss weight;

[0042] Attribute regression loss function Where i is the index of the summation process, ranging from 1 to n β , n β Sample size, z c is the basic real attribute, is the embedding attribute score;

[0043] Attribute cross entropy loss function Where C s is the visible class in all class samples C, c is a real class label, is the predicted label type, represents the set of all possible categories;

[0044] Debiasing loss function In the formula and σ s represents the mean and variance of the prediction scores of the visible classes, and σ uRepresents the mean and variance of the prediction scores for the unseen classes.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] This method acquires global visual features and semantic features, extracts features from different dimensions, and fuses them. This feature fusion method effectively integrates features from different sources, enhancing the generalization capability of image classification systems and improving recognition performance for zero-shot classification tasks. By operating on features in a semantic embedding space, the method can use the attributes of visible classes to infer the attributes of unseen classes, thereby improving the accuracy and generalization of classification tasks.

[0047] The image classification system of the present invention is designed with a feature extraction module, a feature fusion module FFT, a feature interaction module FIN and an optimization module. The feature extraction module extracts significant local features, mixed features, global features and local features. Feature fusion is achieved through the feature fusion module, which reduces the dependence on a single feature source and improves the robustness under different test benchmarks. The optimization module combines the attribute regression loss function, the attribute cross entropy loss function and the debiasing loss function to form a comprehensive loss function. The attribute regression loss function enables the system to accurately map visual features to corresponding semantic embeddings, and the attribute cross entropy loss function maximizes the similarity between the image embedding and the semantic vector of its corresponding category, while minimizing the similarity with the non-corresponding category, thereby improving the system's classification accuracy for the category. The debiasing loss function effectively solves the problem of bias that may arise between visible and invisible classes during the training process, thereby promoting fair learning and effective prediction of invisible classes. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is the overall framework diagram of the present invention;

[0049] Figure 2 is the hyperparameter on the CUB dataset The effect diagram of the value;

[0050] Figure 3 is the hyperparameter on the CUB dataset The effect diagram of the value;

[0051] Figure 4 This is a qualitative result diagram of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] like Figure 1As shown, the image classification method based on zero-sample learning described in this embodiment includes the following steps:

[0054] Step 1: Build an image classification system, which includes a feature extraction module, a feature fusion module FFT, a feature interaction module FIN and an optimization module

[0055] Among them, the feature extraction module is used to extract global visual features and semantic features. The feature fusion module extracts features of different dimensions in the visual features and semantic features, and fuses them to generate fused features containing multiple high dimensions. The feature interaction module maps the fused visual features obtained from the feature fusion module to the semantic embedding space. In order to make the system achieve the optimal state in multiple aspects, the optimization module reduces the dependence on the distribution of training data while learning a more generalized feature representation. Attribute cross entropy loss function And the debiasing loss function Combine to form a comprehensive loss function Where λ AR is the attribute regression loss weight, λ DEB is the debiasing loss weight.

[0056] Introduced attribute regression loss function To constrain the system, accurately map visual features to corresponding semantic embeddings. Visual semantic mapping is regarded as a regression problem, and minimizes a set of samples of size n β The basic real property z c and embedding attribute scores The mean square error between them is used to obtain the attribute regression loss function:

[0057]

[0058] Where i is the index of the summation process, ranging from 1 to n β .

[0059] Attribute regression loss function The main purpose of the algorithm is to ensure that the system can accurately predict the existence probability of each attribute in the image. By minimizing the gap between the predicted score and the true attribute value, the visual feature map learned by the system can better reflect the semantic attributes of the image.

[0060] Since when images have visually relevant attributes, the relevant image embedding will be projected near its semantic vector z^{c}, attribute-based cross entropy loss is adopted To optimize the parameters of the system. Given a batch of n β training images and its corresponding class semantic vector z c , define the attribute cross entropy loss function for:

[0061]

[0062] Among them C s is the visible class in all category samples C, is the predicted label type, Represents the set of all possible categories.

[0063] Attribute cross entropy loss function It can promote the system's sensitivity to subtle differences between different categories, which is especially important when distinguishing similar categories. It also guides the system to distinguish the semantic attributes of different categories. It improves the system's classification accuracy by maximizing the similarity between the image embedding and the semantic vector of its corresponding category, while minimizing the similarity with non-corresponding categories. Specifically, it first calculates the dot product between the visual embedding and each class semantic vector to generate the class logits. This operation is performed to maximize the compatibility score between the image and its corresponding class semantic vector.

[0064] A major challenge in zero-shot classification tasks is to avoid biasing the system towards the visible classes in the training data, because there are only visible classes in the training process, which leads to a significantly higher accuracy of the visible classes than the unseen classes during testing. Therefore, in addition to the attribute regression loss and attribute cross entropy loss mentioned above, a debiasing loss function is introduced. This function reduces this bias by ensuring that the system's predicted distributions for seen and unseen classes are consistent, thereby improving the system's generalization ability for unseen classes. as follows:

[0065]

[0066] in, and σ s represents the mean and variance of the prediction scores of the visible classes, and σ u Represents the mean and variance of the prediction scores for the unseen classes.

[0067] Debiasing loss function Pursuing distribution consistency from the perspective of mean and variance can better balance the score dependencies between the seen and unseen classes. Moreover, by minimizing the distribution difference of the mean and variance of the predicted scores between the two domains, the debiased loss function It helps to improve the system's generalization ability for unseen categories.

[0068] Step 2: Use the visible class dataset as the training set and the invisible class dataset as the test set, and obtain visual global features and semantic features through CNN visual backbone and GloVe

[0069] In generalized zero-shot learning, the training data of the visible class is used Total C seen categories, of which Represents the visible class training sample i, Y i s Represents the category corresponding to the visible class training sample i. There is also a dataset of invisible classes Represents the unseen class training sample j, Y i u Represents the category corresponding to the invisible class training sample i, and there are C invisible classes in total unseen This part of the data does not participate in training and is only used to evaluate performance in the testing phase.

[0070] There exists a category c∈C seen ∪C unseen = C, whose semantic vector set is z containing attribute α c ={z1,z2,…,z α Semantic vector sets are used to describe different attributes. By using semantic information to build relationships between visible and invisible classes, knowledge can be transferred from visible classes to invisible classes.

[0071] According to the attribute description in the semantic vector set, the language model GloVe is used to extract the attribute name f from each attribute. sem The semantic attribute information F is learned from sem . Using CNN from the original feature f cv Visual feature extraction global feature F g , the formula is as follows:

[0072] F g ,F sem =CNN(f cv ),GloVe(f sem ).

[0073] Step 3: Extract features of different dimensions from visual features and semantic features through the feature extraction module, and fuse them in the feature fusion module FFT to generate fused visual features containing multiple high dimensions.

[0074] Features of different dimensions include: global features, local features, mixed features, and significant local features. The generation of fused visual features includes the following steps:

[0075] (1) Global feature extraction

[0076] Global feature Fg The extraction method is the same as that used in step 2.

[0077] (2) Combining visual features with semantic features to enhance the expressive power of features and enable classification of unseen categories;

[0078] Specifically, the dot product operation of visual features and semantic features is performed to generate the interaction feature α, and then the interaction feature α is normalized to generate the feature weight A, so as to convert the interaction feature into a weight for subsequent feature calculation. The formula is:

[0079]

[0080] where α i is the i-th element of the input interaction feature α in the last dimension, j is the index of the sum over all categories, and the sum is the sum of all elements in the last dimension.

[0081] (3) Local feature extraction

[0082] Use feature weight A to adjust global feature F g Perform weighting to generate new weighted visual features, namely new high-level local features F l , better capture the details of the image, the formula is as follows:

[0083]

[0084] Where r is the index of the summation process, ranging from 1 to R, and R is the size of the last dimension of the feature.

[0085] (4) Hybrid feature extraction

[0086] The feature weights used to allow the image classification system to dynamically balance global features and local features are introduced, that is, to calculate the global feature F g With local features F l By introducing the feature weight λ, the system can better generalize to new categories. This is because the weight allows the system to identify and utilize key features that are shared between different categories and help classification. The feature weight λ is then used to adjust the global feature F g With local features F l Perform weighted summation to generate mixed features F mix , so that the system can balance between different features and combine the similarity and importance between features to provide richer information for zero-shot classification. mix The formula is as follows:

[0087] F mix =λ·F g+(1-λ)·F l

[0088] (5) Extraction of salient local features

[0089] By selecting the most representative features, we can avoid paying too much attention to irrelevant features during training, thereby better capturing the essential characteristics of the data.

[0090] Specifically, by calculating the global feature F g and local features F l The element-wise maximum value of max , which can also be thought of as selecting the maximum response at each location. F max The calculation formula is as follows:

[0091] F max =max(R g ,F l )

[0092] Salient local features F max It helps to retain features that are highly relevant to the data and remove irrelevant features. At the same time, selecting in the feature space can reduce the number of features that the system needs to process, thereby improving computational efficiency.

[0093] (6) Feature Fusion

[0094] In order to provide the system with a comprehensive feature representation so that it can capture the unique information of different feature sources, enhance the system's recognition ability of unseen classes, reduce dependence on a single feature source, and improve the robustness of features, the global feature F g , local features F l , mixed features F mix and the salient local features F max The final feature representation F is obtained by fusion. The formula is as follows:

[0095]

[0096] in is the fusion coefficient. In order to achieve the best feature fusion effect, the value of n is determined by trial and error.

[0097] By fusing these features, we can smooth out the extreme differences between different features while retaining their comprehensive information, reducing the system's over-reliance on a single feature. Furthermore, this fusion strategy enables the system to exhibit better adaptability when faced with diverse data distributions.

[0098] Step 4: In the feature interaction module FIN, the fused visual features are mapped to the semantic space through the mapping function to obtain the semantic embedding of the corresponding visual features. Then optimize the system through the optimization module

[0099] In order to maximize the ability of the fusion feature F, it is combined with the semantic feature F sem To enhance the system's ability to identify categories. Specifically, after generating the fusion feature, the semantic feature F sem Semantic attribute vector in As support vector. Based on a mapping function M, the local enhanced visual feature F is combined with the semantic attribute information V α Matching. The fused visual features F are mapped to the semantic attribute information V through the mapping function M. α Consistent semantic embedding space, get the embedding vector mapped to the semantic attribute space Where W is the embedding matrix that embeds the fusion feature F into the semantic attribute space, is the semantic attribute vector V α The transpose of .

[0100] Essentially, Represents an attribute score, representing image x i The confidence of the attribute a, that is, the degree of matching or similarity. Given a set of semantic attribute vectors The mapped semantic embedding matrix can be obtained

[0101] This mapping process not only ensures that the feature representation is closely aligned with the semantic attributes of the category, but also provides a method for accurate classification in high-dimensional semantic space through the transformation of the embedding matrix W. Quantized image x i The confidence of each attribute in provides a direct basis for attribute-based category inference.

[0102] By operating on features in the semantic embedding space, the attribute information of the seen classes can be used to infer the attributes of the unseen classes, thereby improving the accuracy and generalization ability of the classification task.

[0103] By embedding semantic The loss function is input to optimize the system so that it can accurately predict the attributes of the seen classes, effectively classify the unseen categories, and reduce the bias caused by inconsistent data distribution.

[0104] Using loss function The difference between the system's prediction and the true label is calculated to get the loss value. This value quantifies the system's performance under the current parameters. The loss value is then used to backpropagate the parameters and calculate the loss function. The gradient of each parameter. Then use the optimization algorithm to update the system parameters according to the gradient. The learning rate is a hyperparameter that controls the step size of each update. Repeat the above steps until the performance of the system on the validation set no longer improves or the preset number of iterations is reached. Then use the test set to evaluate the performance of the system. The test set should be completely independent of the training process and is used to simulate the performance of the system in actual applications. Based on the performance on the validation set and test set, it may be necessary to adjust the hyperparameters (such as learning rate, batch size, number of iterations, etc.) to optimize the performance. Once the training is completed and the performance is satisfactory, it can be deployed to the actual application.

[0105] Step 5: After training is completed, first obtain the test instance x i Semantic embedding in semantic space Then use nearest neighbor matching and explicit self-calibration to predict x i Tags c * ;

[0106]

[0107] where c * represents the predicted label, C u / C correspond to zero-shot image classification under CZSL / GZSL settings, respectively. c is the semantic embedding of category c, II is the indicator function;

[0108] By computing the semantic embedding of the test sample and category semantic vector z c The dot product represents the compatibility or similarity score between the sample feature and the category semantics; Finally, for each test sample x i , the system will select the category c with the highest compatibility score as the predicted category.

[0109] The present invention provides a computer device, comprising a memory and a processor, wherein the memory is used to store computer execution instructions, the processor and the memory are connected via a bus, and when the computer device is running, the processor executes the steps of the image classification method stored in the memory.

[0110] The present invention provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer can execute the steps of the image classification method.

[0111] Experimental part:

[0112] (1) Image classification effect verification

[0113] To comprehensively evaluate the performance of the proposed image classification system on datasets of varying complexity, experiments were conducted on three widely recognized ZSL benchmark datasets: CUB, AWA2, and SUN. These datasets cover a range of classifications from fine-grained to coarse-grained, providing diverse testing scenarios.

[0114] The CUB dataset contains 11,788 images of 200 bird species. The dataset is divided into 150 visible classes and 50 unseen classes. 312 attributes are used to describe the semantic information of the images. These attributes cover fine-grained features of birds, such as color and shape, providing a rich resource for fine-grained classification tasks.

[0115] The AWA2 dataset contains 37,322 images classified into 50 animal categories, 40 of which are seen and 10 are unseen. The dataset has 85 attributes describing the properties of animals, providing information about animal images and their corresponding attributes.

[0116] The SUN dataset is more diverse, containing 717 scene categories and 14,340 images. The classification task in this dataset is more complex, with 645 categories considered as seen classes and 72 categories as unseen classes. Each category has 102 attribute descriptions, focusing on scene classification and attribute recognition. This dataset provides a large number of scene images and attribute annotations, making it well-suited for scene recognition and attribute learning tasks.

[0117] Two evaluation settings are adopted: conventional ZSL (CZSL) and generalized ZSL (GZSL).

[0118] In the CZSL setting, the system's ability to classify unseen classes is evaluated by first computing the average accuracy for each class independently and then dividing the cumulative sum by the number of classes; that is, the system's performance is measured by computing the top-1 accuracy of the test sample, denoted as acc.

[0119] In the GZSL setting, classification performance for both the seen and unseen classes is evaluated. The average per-class top-1 accuracy for the seen classes is calculated, denoted as S, and the average per-class top-1 accuracy for the unseen classes is calculated, denoted as U. In addition to calculating the accuracy for both separately, the harmonic mean of the two, H, is used as a comprehensive evaluation metric to balance the classification performance for both seen and unseen classes.

[0120]

[0121] Among them, U represents the classification accuracy of the unseen class, and S represents the classification accuracy of the seen class.

[0122] ResNet101 pre-trained on ImageNet is used as the CNN backbone to extract feature maps. For other modules in the system that need to be trained, the SGD optimizer with hyperparameters is used for optimization, where the momentum is set to 0.9, the weight decay is set to 0.0001, the learning rate and the batch size are set to 0.0001 and 50 respectively. AR Set to 0.003, λ DEB Set to 0.2.

[0123] About Lambda AR Numerical setting of attribute regression loss: Attribute regression loss is used to measure the importance of attribute regression loss and guide the system to perform effective visual-semantic interaction. It aims to ensure that the system can accurately predict the attributes of objects in the image, such as color, size, shape, etc. The attribute regression loss weight λ is set on the CUB dataset. AR The value of is studied, such as Figure 2 As shown, The value of is: when When the setting is small, the accuracy of the unseen class U and the harmonic mean H tend to increase. This is because the attribute regression loss value is too small to guide the system to fully learn the attribute features, failing to achieve the maximum correction effect on the system, resulting in the system's inaccurate attribute prediction. AR When the setting is large, all evaluation indicators tend to decrease. This is because the loss value of attribute regression loss is too large, which makes the system focus too much on the attribute regression task and ignore other important loss functions. At the same time, considering the accuracy of the unseen class U, the visible class S and the harmonic average H, when When the value is set to 0.003, our system achieves the most balanced performance, which shows that under this value, the attribute regression loss can properly balance the learning process of the system, neither ignoring the contribution of other loss functions nor effectively guiding the system to learn attribute features. Moreover, under the traditional zero-shot experimental setting, Setting it to 0.003 has little effect on the experimental results. Therefore, λ in all datasets AR Set to 0.003.

[0124] About Lambda DEB Numerical setting: λ DEE It is a weight used to control the effect of debiasing loss. The appropriate value is conducive to the debiasing loss function to balance the bias problem between visible and invisible classes. DEB The value of is studied, such as Figure 3 As shown, λ DEB The value of is: DEB ={0.001,0.01,0.1,0.2,0.3,0.4}. When increasing λDEB When the value of , the accuracy of the unseen class increases, and the accuracy of the seen class decreases. DEE When the setting is small, the accuracy of the unseen class is significantly lower than that of the seen class. This is because the debiasing loss value is too small to effectively balance the system's bias towards the seen class. BEB When the value is set to a large value, the accuracy of the invisible class is significantly higher than that of the visible class. This is because the debiasing loss value is too large, which overemphasizes the results of the invisible class, resulting in an imbalance in the accuracy of the visible and invisible classes. When set to 0.2, the system can also achieve the highest value in the traditional zero-shot classification setting, so the debiasing loss weight λ in all datasets is set to DEB Set to 0.2.

[0125] Randomly select some data from the visible and invisible classes of the CUB dataset, and use the trained system to evaluate the selected data for qualitative analysis. Select the top five attributes with the highest evaluation scores for each sample, such as Figure 4 As shown in the figure, these attribute descriptions and the sample images that match them are displayed. On the visible class side, each sample has at least four attribute descriptions that closely match the characteristics of the bird in the image. For example, in the sample "Lvory Gull", its first five attribute descriptions all correspond to the characteristics of the bird in the sample image. On the invisible class side, each sample has at least three attribute descriptions that can match the description of the bird in the sample image. Therefore, the image classification method of the present invention achieves significant performance in attribute recognition of both visible and invisible classes.

[0126] The method of the present invention is compared with other methods on three datasets: CUB, AWA2, and SUN. The results are shown in Table 1.

[0127] Table 1 Comparative experimental results on three datasets

[0128]

[0129] In the traditional zero-shot classification task, the method of the present invention achieved the best accuracy (acc) of 76.8% and 64.4% in CZSL on the CUB and SUN datasets, respectively. These results highlight the system's ability to handle different coarse and fine granularity classifications. On the dataset AWA2, the accuracy (acc) of the present invention in CZSL also showed a competitive performance of 65%. Compared with other methods that performed best on CUB and SUN, the accuracy (acc) of the present invention in CZSL achieved a significant advantage of 4.8% and 0.6% over the advanced methods, respectively. This shows that the present invention provides a comprehensive, robust and generalizable feature space for zero-shot classification tasks to obtain better resolution.

[0130] In the generalized zero-shot classification task, the harmonic mean (H) of our invention surpassed the best-performing method on CUB by 0.2%, and achieved competitive results on the AWA2 dataset. In CUB, TCN had the smallest accuracy difference between the visible class S and the unseen class U, at 0.6%, while our method had a mere 0.2% difference between the visible and unseen classes, significantly exceeding TCN's accuracy. This comparison demonstrates our invention's superiority in balancing the visible and unseen classes, highlighting its significant potential in ZSL tasks.

[0131] In AWA2, the difference in accuracy between the visible class S and the invisible class U is particularly pronounced, generally ranging from 15% to 20%. Even more extreme is AREN, where the difference between the visible class S and the invisible class U reaches 77.3%. Although the balance effect of our method on AWA2 is not optimal, it achieves higher accuracy than f-CLSWGAN, which achieves the best balance. These results indicate that our image classification method has great potential in generalized zero-shot classification tasks.

[0132] (2) Ablation experiment

[0133] In order to further understand the effectiveness of each module and feature of the system, an ablation study was conducted to evaluate the feature fusion FFT, local feature FFT in FFT, and the performance of the proposed system on the CUB dataset. l , significant local features F in FFT max , the mixed feature F in FFT mix , attribute regression loss function and debiasing loss function The effect is shown in Table 2.

[0134] Table 2 Ablation experiments on the CUB dataset

[0135]

[0136] When the feature fusion FFT is not used, the performance of the system is significantly lower than that of the system of the present invention, and the accuracy acc and harmonic average H decrease by 14.4% and 12.7% respectively. l , salient local features F max and mixed features F mix The incomplete system obtains inferior results compared with the system of the present invention, in which there is no local feature F lThe accuracy of the experiment with α decreased most significantly, with the accuracy acc and harmonic mean H decreasing by 12.1% and 9.4% respectively. When using FFT without local features, the system's performance on the dataset dropped sharply because local features help highlight visual features that are more relevant to semantic features.

[0137] Debiasing loss can effectively alleviate the bias problem. When the debiasing loss function is removed It can be seen that the system is obviously more biased towards the visible class, and the difference between the accuracy of the visible class S and the accuracy of the invisible class U reaches 20.2%. After that, the harmonic mean H is increased by 6.7% and the influence of bias is largely eliminated. The performance is further improved by guiding the system to perform effective visual-semantic mapping.

Claims

1. An image classification method based on zero-shot learning, characterized in that: The following steps are involved: Step 1: Build an image classification system, which includes a feature extraction module, a feature fusion module FFT, a feature interaction module FIN and an optimization module; Step 2: Use the visible class dataset as the training set and the unseen class dataset as the test set, and obtain visual global features and semantic features through the CNN visual backbone and GloVe; Step 3: Extract features of different dimensions from visual features and semantic features through the feature extraction module, and fuse them in the feature fusion module FFT to generate a fused visual feature containing multiple high dimensions; the features of different dimensions include global features, local features, mixed features, and significant local features; Step 4: In the feature interaction module FIN, the fused visual features are mapped to the semantic space through the mapping function to obtain the semantic embedding of the corresponding visual features. Then optimize the system through the optimization module; Step 5: After training is completed, first obtain the test instance x i Semantic embedding in semantic space Then use nearest neighbor matching and explicit self-calibration to predict x i Tags c * ; where c * represents the predicted label, C u / C correspond to zero-shot image classification under CZSL / GZSL settings, respectively. c is the semantic embedding of category c, II is the indicator function; By calculating the semantic embedding of the test sample and category semantic vector z c The dot product represents the compatibility or similarity score between the sample feature and the category semantics; Finally, for each test sample x i , the system will select the category c with the highest compatibility score as the predicted category; The process of obtaining the visual global features and semantic features described in step 2 is as follows: In generalized zero-shot learning, the training data of the visible class is used Total C seen categories, of which Represents training sample i, Y i s Represents the category corresponding to sample i; there is also a dataset of unseen classes Represents the unseen class training sample i, Y i u Represents the category corresponding to the unseen class training sample i; there is a category c∈C seen ∪C unseen = C, whose semantic vector set is z containing attribute α c ={z1,z2,…,z α }; According to the attribute description in the semantic vector set, the language model GloVe is used to extract the attribute name f from each attribute. Sem The semantic attribute information F is learned from sem ; Use CNN to extract the original features f cv Visual feature extraction global feature F g , the formula is as follows: F g ,F sem =CNN(f cv ),GloVe(f sem ); The generation of the fused visual features in step 3 includes the following steps: (1) Global feature extraction: Use step 2 to extract the global feature F g Methods; (2) Combining visual features with semantic features to enhance the expressive power of features and enable classification of unseen categories; The interaction between visual features and semantic features is calculated to generate the interaction feature α, which is then normalized to generate the feature weight A, so as to convert the interaction feature into a weight for subsequent feature calculation. The formula is: where α i is the i-th element of the input interaction feature α in the last dimension, j is the index of the sum of all categories, and the sum is the sum of all elements in the last dimension; (3) Local feature extraction Use feature weight A to adjust global feature F g Perform weighting to generate new weighted visual features, namely new high-level local features F l , better capture the details of the image, the formula is as follows: Where r is the index of the summation process, ranging from 1 to R, and R is the size of the last dimension of the feature; (4) Hybrid feature extraction Calculate the global feature F g With local features F l The feature weight λ between them is then used to adjust the global feature F g With local features F l Perform weighted summation to generate mixed features F mix , mixed feature F mix The formula is as follows: F mix =λ·F g +(1-λ)·F l (5) Extraction of salient local features By calculating the global feature F g and local features F l The element-wise maximum value of max , F max The calculation formula is as follows: F max =max(F g ,F l ) (6) Feature Fusion The global feature F g , local features F l , mixed features F mix and the salient local features F max The final feature representation F is obtained by fusion, and the formula is as follows: in is the fusion coefficient.

2. The image classification method based on zero-shot learning according to claim 1, characterized in that: Semantic embedding as described in step 4 The acquisition process is: using semantic feature F sem Semantic attribute vector in As a support vector, based on a mapping function M, the local enhanced visual features F are combined with the semantic attribute information V α Matching, mapping the fused visual features F to the semantic attribute information V through the mapping function M α Consistent semantic embedding space, the output of the feature interaction module FIN, that is, the embedding vector obtained by mapping to the semantic attribute space Where W is the embedding matrix that embeds the fusion feature F into the semantic attribute space, is the semantic attribute vector V α The transpose of .

3. The image classification method based on zero-shot learning according to claim 2, characterized in that: The optimization module in step 4 optimizes the system by introducing the loss function To achieve this, the loss function is in is the attribute regression loss function, λ AR is the attribute regression loss weight, is the attribute cross entropy loss function, is the debiasing loss function, λ DEB is the debiasing loss weight; Attribute regression loss function Where i is the index of the summation process, ranging from 1 to n β , n β Sample size, z c is the basic real attribute, is the embedding attribute score; Attribute cross entropy loss function C s is the visible class in all class samples C, c is a real class label, is the predicted label type, represents the set of all possible categories; Debiasing loss function In the formula and σ s represents the mean and variance of the prediction scores of the visible class, and σ u Represents the mean and variance of the prediction scores for the unseen classes.

Images