Image classification method based on zero sample learning

Abstract

The invention discloses an image classification method based on zero sample learning, and the method comprises the following steps: firstly building an image classification system, employing a visible data set as a training set, employing an invisible data set as a test set, and obtaining visual global and semantic features through a CNN visual backbone and GloVe; and then, extracting different dimension features from the visual and semantic features and fusing the different dimension features into multivariate high-dimension fused visual features. Then, the fused visual features are mapped to a semantic space through a mapping function, and semantic embedding is obtained and optimized through an optimization module; after training is completed, test instance semantic embedding is obtained, and by means of nearest neighbor matching and explicit self-calibration prediction labels, the category with the highest compatibility score is selected as the prediction category by calculating the test sample semantic embedding and category semantic vector dot product. According to the method, the problem of insufficient consideration of different dimension features in zero sample classification can be solved, the prediction distribution consistency of visible and invisible classes is ensured, and the generalization ability of the invisible classes is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and in particular relates to an image classification method based on zero-sample learning. Background Art

[0002] Zero-shot learning (ZSL) is a new deep learning method that aims to break through the dependence on label annotations and solve the problem of unknown data recognition. Inspired by the process of human cognition of new things, zero-shot learning mines the intrinsic connection between visible and invisible classes by learning known training data, and realizes the recognition of invisible classes in the reasoning stage. Zero-shot learning uses semantic information as a bridge to connect visible and invisible classes, extracts semantically corresponding attribute information from visible class training data, and extends it to the recognition and classification of invisible classes, enhancing cross-class generalization capabilities to accurately identify invisible classes. Zero-shot learning requires careful design of algorithms in the training stage to make the visual image information of visible classes correspond to semantic information, and consider generalization of invisible classes. In the testing stage, an ideal zero-shot learning model can find the most relevant semantics as attribute prediction for invisible class instances. There are two key problems in zero-shot learning: one is how to associate information of two different modalities, namely, visual and semantic information; the other is that the visual and semantic information of visible classes is prone to overfitting in the training stage, resulting in the inference results of the model biased towards visible classes when facing invisible classes in the testing stage.

[0003] Early zero-shot learning methods first embed the features of the visible class into the semantic space and then classify the unseen class. Since these models are based on Bayesian probability prediction and overly rely on labeled data, overfitting problems often occur in zero-shot learning. To solve this problem, some models began to use global features, but using only global features cannot accurately capture local features that directly correspond to semantic attributes. For example, if the distinguishing feature of a category is eye color, the global feature reflects more of the overall appearance rather than local details. Directly using global feature embedding may not accurately reflect this distinguishing attribute, resulting in failure to achieve the desired effect.

[0004] Due to the limitations of global visual features, the mapping relationship between visual and semantic features is not clear enough. Therefore, some methods combine the attention mechanism with the model to locate local image attributes, but lack the relationship representation between local features. Later, some methods further used regional graph learning to enhance the local reasoning of visual features, improving the model's ability in visual semantic interaction. However, these models still have the following defects: First, the attention mechanism integrated in the model cannot effectively locate local features, and thus cannot effectively represent local features; second, these methods have bias problems when dealing with visible and invisible classes. Summary of the invention

[0005] The purpose of the present invention is to provide an image classification method based on zero-shot learning, which can solve the problem of insufficient consideration of features of different dimensions in zero-shot classification, ensure the consistency of predicted distribution of visible and invisible classes, and improve the generalization ability of invisible classes.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] An image classification method based on zero-shot learning comprises the following steps:

[0008] Step 1: Building an image classification system, the system includes a feature extraction module, a feature fusion module FFT, a feature interaction module FIN and an optimization module;

[0009] Step 2: Use the visible class dataset as the training set and the invisible class dataset as the test set, and obtain visual global features and semantic features through CNN visual backbone and GloVe;

[0010] Step 3: extracting features of different dimensions from visual features and semantic features through the feature extraction module, and fusing them in the feature fusion module FFT to generate fused visual features containing multiple high dimensions; the features of different dimensions include global features, local features, mixed features and significant local features;

[0011] Step 4: In the feature interaction module FIN, the fused visual features are mapped to the semantic space through a mapping function to obtain the semantic features of the corresponding visual features. Then the system is optimized through the optimization module;

[0012] Step 5: After training is completed, first obtain the test instance x i Semantic embedding in semantic space Then use nearest neighbor matching and explicit self-calibration to predict x i Tags * ;

[0013]

[0014] where c * represents the predicted label, C u / C correspond to zero-shot image classification under CZSL / GZSL settings, respectively. c is the semantic embedding of category c, and II is the indicator function;

[0015] By calculating the semantic embedding of the test sample and the category semantic vector z c The dot product represents the compatibility or similarity score between the sample feature and the category semantics; finally, for each test sample x i, the system will select the category c with the highest compatibility score as the predicted category.

[0016] Furthermore, the process of obtaining the visual global features and semantic features in step 2 is as follows:

[0017] In generalized zero-shot learning, the training data of the visible class is used Total C seen categories, of which represents training sample i, Y i s Represents the category corresponding to sample i; there is also a data set of unseen classes Represents the invisible class training sample i, Y i u Represents the category corresponding to the invisible class training sample i; there is a category c∈C seen ∪C unseen = C, whose semantic vector set is z containing attribute α c ={z1,z2,…,z α};

[0018] According to the attribute description in the semantic vector set, the language model GloVe is used to extract the attribute name f from each attribute. sem The semantic attribute information F is learned from sem ; Use CNN to extract the original feature f cv Visual feature extraction global feature F g , the formula is as follows:

[0019] F g ,F sem =CNN(f cv ),GloVe(f sem ).

[0020] Furthermore, the generation of the fused visual features in step 3 includes the following steps:

[0021] (1) Global feature extraction: Extract the global feature F from step 2 g Methods;

[0022] (2) Combining visual features with semantic features to enhance the expressive power of features and to classify unseen categories;

[0023] The interaction between visual features and semantic features is calculated to generate the interaction feature α, and then the interaction feature α is normalized to generate the feature weight A, so as to convert the interaction feature into a weight for subsequent feature calculation. The formula is:

[0024]

[0025] where αi is the i-th element of the input interaction feature α in the last dimension, j is the index of the sum of all categories, and the sum is the sum of all elements in the last dimension;

[0026] (3) Local feature extraction

[0027] Use feature weight A to adjust global feature F g Weighted to generate new weighted visual features, namely new high-level local features F l , better capture the details of the image, the formula is as follows:

[0028]

[0029] Where r is the index of the summation process, ranging from 1 to R, and R is the size of the last dimension of the feature;

[0030] (4) Hybrid feature extraction

[0031] Calculate the global feature F g With local features F l The feature weight λ between them is then used to adjust the global feature F g With local features F l Perform weighted summation to generate mixed features F mix , mixed feature F mix The formula is as follows:

[0032] F mix =λ·F g +(1-λ)·F l

[0033] (5) Extraction of salient local features

[0034] By calculating the global feature F g and local features F l The element-wise maximum value of max , F max The calculation formula is as follows:

[0035] F max =max(F g ,F l )

[0036] (6) Feature Fusion

[0037] The global feature F g , local features F l , mixed features F mix And the significant local features F max The final feature representation F is obtained by fusion, and the formula is as follows:

[0038]

[0039] in is the fusion coefficient.

[0040] Furthermore, the semantic embedding described in step 4 The process of obtaining is: using semantic feature F sem The semantic attribute vector in As a support vector, based on a mapping function M, the local enhanced visual feature F is combined with the semantic attribute information V α Matching, through the mapping function M, the fused visual feature F is mapped to the semantic attribute information V α Consistent semantic embedding space, the output of the feature interaction module FIN, that is, the embedding vector mapped to the semantic attribute space Where W is the embedding matrix that embeds the fusion feature F into the semantic attribute space, is the semantic attribute vector V α The transpose of .

[0041] Furthermore, the optimization module described in step 4 optimizes the system by introducing a loss function To achieve this, the loss function is in is the attribute regression loss function, λ AR is the attribute regression loss weight, is the attribute cross entropy loss function, is the debiasing loss function, λ DEB is the debiasing loss weight;

[0042] Attribute regression loss function Where i is the index of the summation process, ranging from 1 to n β , n β Sample size, z c is the basic real attribute, is the embedding attribute score;

[0043] Attribute Cross Entropy Loss Function Where C s is the visible class in all class samples C, c is a real class label, is the predicted label type, represents the set of all possible categories;

[0044] Debiasing loss function In the formula and σ s represents the mean and variance of the prediction scores of the visible classes, and σ uRepresents the mean and variance of the prediction scores for the unseen classes.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention obtains visual global features and semantic features, and extracts features of different dimensions for fusion. This feature fusion method effectively integrates features from different sources, which not only enhances the generalization ability of the image classification system, but also improves the recognition performance of zero-sample classification tasks. By operating features in the semantic embedding space, the present invention can use the attribute information of the visible class to infer the attributes of the invisible class, thereby improving the accuracy and generalization ability of the classification task.

[0047] The image classification system of the present invention is designed with a feature extraction module, a feature fusion module FFT, a feature interaction module FIN and an optimization module. The feature extraction module extracts significant local features, mixed features, global features and local features. Feature fusion is achieved through the feature fusion module, which reduces the dependence on a single feature source and improves the robustness under different test benchmarks. The optimization module combines the attribute regression loss function, the attribute cross entropy loss function and the debiasing loss function to form a comprehensive loss function. The attribute regression loss function enables the system to accurately map visual features to corresponding semantic embeddings, and the attribute cross entropy loss function maximizes the similarity between the image embedding and the semantic vector of its corresponding category, while minimizing the similarity with the non-corresponding category, so as to improve the classification accuracy of the system to the category. The debiasing loss function effectively solves the problem of bias that may be generated in the training process for visible and invisible classes, thereby promoting fair learning and effective prediction of invisible classes. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is the overall framework diagram of the present invention;

[0049] Figure 2 is the hyperparameter on the CUB dataset The effect diagram of the value of ;

[0050] Figure 3 is the hyperparameter on the CUB dataset The effect diagram of the value of ;

[0051] Figure 4 This is a qualitative result diagram of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] like Figure 1As shown, the image classification method based on zero-sample learning described in this embodiment includes the following steps:

[0054] Step 1: Build an image classification system, which includes a feature extraction module, a feature fusion module FFT, a feature interaction module FIN and an optimization module

[0055] The feature extraction module is used to extract global visual features and semantic features. The feature fusion module extracts features of different dimensions in the visual features and semantic features, and fuses them to generate fused features containing multiple high dimensions. The feature interaction module maps the fused visual features obtained from the feature fusion module to the semantic embedding space. In order to achieve the optimal state of the system in multiple aspects, the optimization module reduces the dependence on the distribution of training data while learning a more generalized feature representation. Attribute Cross Entropy Loss Function And the debiasing loss function Combine to form a comprehensive loss function Where λ AR is the attribute regression loss weight, λ DEB is the debiasing loss weight.

[0056] Introduced attribute regression loss function To constrain the system, accurately map visual features to corresponding semantic embeddings. Visual semantic mapping is regarded as a regression problem, and minimizes a set of samples of size n β The basic real property z c With embedded attribute scores The mean square error between them is used to obtain the attribute regression loss function:

[0057]

[0058] Where i is the index of the summation process, ranging from 1 to n β .

[0059] Attribute regression loss function The main purpose of is to ensure that the system can accurately predict the existence probability of each attribute in the image. By minimizing the gap between the predicted score and the true attribute value, the visual feature map learned by the system can better reflect the semantic attributes of the image.

[0060] Since when images have visually relevant attributes, the relevant image embedding will be projected near its semantic vector z^{c}, the attribute-based cross entropy loss is adopted To optimize the parameters of the system. Given a batch of n β training images and its corresponding class semantic vector z c , define the attribute cross entropy loss function for:

[0061]

[0062] Among them C s is the visible class in all class samples C, is the predicted label type, Represents the set of all possible categories.

[0063] Attribute Cross Entropy Loss Function It can promote the system's sensitivity to subtle differences between different categories, which is especially important when distinguishing similar categories. It can also guide the system to distinguish the semantic attributes of different categories. It improves the system's classification accuracy by maximizing the similarity between the image embedding and its corresponding category semantic vector, while minimizing the similarity with non-corresponding categories. Specifically, the dot product between the visual embedding and each class semantic vector is first calculated to generate the class logic. This operation is done to make the image have the highest compatibility score with its corresponding class semantic vector.

[0064] A major challenge in zero-shot classification tasks is to avoid biasing the system towards the visible classes in the training data, because there are only visible classes in the training process, which leads to significantly higher accuracy of the visible classes than the unseen classes during testing. Therefore, in addition to the attribute regression loss and attribute cross entropy loss mentioned above, a debiasing loss function is introduced This function reduces this bias by ensuring that the system's predicted distributions for seen and unseen classes are consistent, thereby improving the system's generalization ability for unseen classes. as follows:

[0065]

[0066] in, and σ s represents the mean and variance of the prediction scores of the visible classes, and σ u Represents the mean and variance of the prediction scores for the unseen classes.

[0067] Debiasing loss function Pursuing distribution consistency in terms of both mean and variance can better balance the score dependencies between the seen and unseen classes. Moreover, by minimizing the distribution differences in the mean and variance of the predicted scores between the two domains, the debiasing loss function It helps to improve the system's generalization ability for unseen categories.

[0068] Step 2: Use the visible class dataset as the training set and the invisible class dataset as the test set, and obtain visual global features and semantic features through CNN visual backbone and GloVe

[0069] In generalized zero-shot learning, the training data of the visible class is used Total C seen categories, of which Represents the visible class training sample i, Y i s Represents the category corresponding to the visible class training sample i. There is also a dataset of invisible classes represents the invisible class training sample j, Y i u Represents the category corresponding to the invisible class training sample i, and there are C invisible classes in total unseen This part of data does not participate in training and is only used to evaluate performance in the test phase.

[0070] There exists a category c∈C seen ∪C unseen = C, whose semantic vector set is z containing attribute α c ={z1,z2,…,z α The semantic vector set is used to describe different attributes. By building the relationship between visible and invisible classes with the help of semantic information, the purpose of transferring knowledge from visible classes to invisible classes is achieved.

[0071] According to the attribute description in the semantic vector set, the language model GloVe is used to extract the attribute name f from each attribute. sem The semantic attribute information F is learned from sem . Using CNN from the original feature f cv Visual feature extraction global feature F g , the formula is as follows:

[0072] F g ,F sem =CNN(f cv ),GloVe(f sem ).

[0073] Step 3: Extract features of different dimensions from visual features and semantic features through the feature extraction module, and fuse them in the feature fusion module FFT to generate fused visual features containing multiple high dimensions.

[0074] Features of different dimensions include: global features, local features, mixed features, and significant local features. The generation of fused visual features includes the following steps:

[0075] (1) Global feature extraction

[0076] Global feature Fg The extraction method is the same as that used in step 2.

[0077] (2) Combining visual features with semantic features to enhance the expressive power of features and to classify unseen categories;

[0078] Specifically, the visual features and semantic features are dot-producted to generate the interaction feature α, and then the interaction feature α is normalized to generate the feature weight A, so as to convert the interaction feature into a weight for subsequent feature calculation. The formula is:

[0079]

[0080] where α i is the i-th element of the input interaction feature α in the last dimension, j is the index over which all categories are summed, and the sum is the sum of all elements in the last dimension.

[0081] (3) Local feature extraction

[0082] Use feature weight A to adjust global feature F g Weighted to generate new weighted visual features, namely new high-level local features F l , better capture the details of the image, the formula is as follows:

[0083]

[0084] Where r is the index of the summation process, ranging from 1 to R, and R is the size of the last dimension of the feature.

[0085] (4) Hybrid feature extraction

[0086] The feature weights used to allow the image classification system to dynamically balance global features and local features are introduced, that is, to calculate the global feature F g With local features F l By introducing the feature weight λ, the system can better generalize to new categories. This is because the weight allows the system to identify and utilize key features that are shared between different categories and help with classification. The feature weight λ is then used to adjust the global feature F g With local features F l Perform weighted summation to generate mixed features F mix In this way, the system can balance between different features and combine the similarity and importance between features to provide richer information for zero-shot classification. mix The formula is as follows:

[0087] F mix =λ·F g+(1-λ)·F l

[0088] (5) Extraction of salient local features

[0089] By selecting the most representative features, we can avoid paying too much attention to irrelevant features during training, thereby better capturing the essential characteristics of the data.

[0090] Specifically, by calculating the global feature F g and local features F l The element-wise maximum value of max , which can also be thought of as selecting the maximum response at each location. max The calculation formula is as follows:

[0091] F max =max(R g ,F l )

[0092] The significant local feature F max It helps to retain the features that are highly relevant to the data and remove irrelevant features. At the same time, selecting in the feature space can reduce the number of features that the system needs to process, thereby improving computational efficiency.

[0093] (6) Feature Fusion

[0094] In order to provide the system with a comprehensive feature representation so that it can capture the unique information of different feature sources, enhance the system's recognition ability for unseen classes, reduce dependence on a single feature source, and improve the robustness of the feature, the global feature F g , local features F l , mixed features F mix And the significant local features F max The final feature representation F is obtained by fusion. The formula is as follows:

[0095]

[0096] in is the fusion coefficient. In order to achieve the best feature fusion effect, the value of n is determined by trial and error.

[0097] By fusing these features, we can smooth out the extreme differences between different features while retaining their comprehensive information, so that the system can reduce its over-reliance on a single feature. Moreover, this fusion strategy can make the system more adaptable when facing diverse data distributions.

[0098] Step 4: In the feature interaction module FIN, the fused visual features are mapped to the semantic space through a mapping function to obtain the semantic embedding of the corresponding visual features. Then optimize the system through the optimization module

[0099] In order to maximize the ability of the fusion feature F, it is combined with the semantic feature F sem To enhance the system's ability to identify categories. Specifically, after generating the fusion feature, the semantic feature F sem The semantic attribute vector in Based on a mapping function M, the local enhanced visual feature F is combined with the semantic attribute information V α Matching. The fused visual feature F is mapped to the semantic attribute information V through the mapping function M. α Consistent semantic embedding space, get the embedding vector mapped to the semantic attribute space Where W is the embedding matrix that embeds the fusion feature F into the semantic attribute space, is the semantic attribute vector V α The transpose of .

[0100] Essentially, represents an attribute score, representing image x i The confidence of the ath attribute, that is, the degree of matching or similarity. Given a set of semantic attribute vectors The mapped semantic embedding matrix can be obtained

[0101] This mapping process not only ensures that the feature representation is closely aligned with the semantic attributes of the category, but also provides a method for accurate classification in high-dimensional semantic space through the transformation of the embedding matrix W. Quantize the image x i The confidence of each attribute in provides a direct basis for attribute-based category inference.

[0102] By operating on features in the semantic embedding space, the attribute information of the seen classes can be used to infer the attributes of the unseen classes, thereby improving the accuracy and generalization ability of the classification task.

[0103] By embedding semantic The loss function is input to optimize the system so that it can accurately predict the attributes of the seen classes, effectively classify the unseen categories, and reduce the bias caused by inconsistent data distribution.

[0104] Using loss function The difference between the system's prediction and the true label is calculated to get the loss value. This value quantifies the system's performance under the current parameters. The loss value is then used to backpropagate the parameters and calculate the loss function The gradient of each parameter. Then use the optimization algorithm to update the system parameters according to the gradient. The learning rate is a hyperparameter that controls the step size of each update. Repeat the above steps until the performance of the system on the validation set no longer improves or the preset number of iterations is reached. Then use the test set to evaluate the performance of the system. The test set should be completely independent of the training process and is used to simulate the performance of the system in actual applications. Depending on the performance on the validation set and test set, it may be necessary to adjust the hyperparameters (such as learning rate, batch size, number of iterations, etc.) to optimize the performance. Once the training is completed and the performance is satisfactory, it can be deployed to the actual application.

[0105] Step 5: After training is completed, first obtain the test instance x i Semantic embedding in semantic space Then use nearest neighbor matching and explicit self-calibration to predict x i Tags * ;

[0106]

[0107] where c * represents the predicted label, C u / C correspond to zero-shot image classification under CZSL / GZSL settings, respectively. c is the semantic embedding of category c, and II is the indicator function;

[0108] By computing the semantic embedding of the test sample and the category semantic vector z c The dot product represents the compatibility or similarity score between the sample feature and the category semantics; finally, for each test sample x i , the system will select the category c with the highest compatibility score as the predicted category.

[0109] The present invention provides a computer device, comprising a memory and a processor, wherein the memory is used to store computer execution instructions, the processor and the memory are connected via a bus, and when the computer device is running, the processor executes the steps of an image classification method stored in the memory.

[0110] The present invention provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer can execute the steps of the image classification method.

[0111] Experimental part:

[0112] (I) Image classification effect verification

[0113] In order to comprehensively evaluate the performance of the proposed image classification system on datasets of different complexity, experiments are conducted on three widely recognized ZSL benchmark datasets: CUB, AWA2, and SUN. These datasets cover category divisions from fine-grained to coarse-grained, providing diverse test scenarios.

[0114] The CUB dataset contains 11,788 images covering 200 species of birds. The dataset is divided into 150 visible classes and 50 invisible classes, of which 312 attributes are used to describe the semantic information of the image. These attributes cover the fine-grained features of birds, such as color, shape, etc., providing rich resources for fine-grained classification tasks.

[0115] The AWA2 dataset contains 37,322 images, divided into 50 animal categories, of which 40 categories are visible and 10 categories are invisible. The dataset has a total of 85 attributes used to describe animal attributes, providing animal images and their corresponding attribute information.

[0116] The SUN dataset is more diverse, containing 717 scene categories and 14,340 images. The classification task of this dataset is more complex, with 645 categories as visible classes and 72 categories as invisible classes. Each category has 102 attribute descriptions, focusing on scene classification and attribute recognition, providing a large number of scene images and attribute annotations, which is very suitable for scene recognition and attribute learning tasks.

[0117] Two evaluation settings are adopted: conventional ZSL (CZSL) and generalized ZSL (GZSL).

[0118] In the CZSL setting, the system’s ability to classify unseen classes is evaluated by first calculating the average accuracy of each class independently, and then dividing its cumulative sum by the number of classes; that is, the system performance is measured by calculating the top-1 accuracy of the test sample, and the accuracy is recorded as acc.

[0119] In the GZSL setting, the classification performance of the visible and invisible classes is evaluated. The average top-1 accuracy of the visible class is calculated as S, and the average top-1 accuracy of the invisible class is calculated as U. In addition to calculating the accuracy of the two separately, the harmonic mean H of the two is also used as a comprehensive evaluation metric to balance the classification performance of the visible and invisible classes.

[0120]

[0121] Among them, U represents the classification accuracy of the unseen class, and S represents the classification accuracy of the seen class.

[0122] ResNet101 pre-trained on ImageNet is used as the CNN backbone to extract feature maps. For other modules in the system that need to be trained, the SGD optimizer with hyperparameters is used for optimization, where the momentum is set to 0.9, the weight decay is set to 0.0001, the learning rate and the batch size are set to 0.0001 and 50 respectively. AR Set to 0.003, λ DEB Set to 0.2.

[0123] About Lambda AR Numerical setting of: Attribute regression loss is used to measure the importance of attribute regression loss and guide the system to perform effective visual-semantic interaction. It aims to ensure that the system can accurately predict the attributes of objects in the image, such as color, size, shape, etc. The attribute regression loss weight λ is set on the CUB dataset. AR The value of is studied, such as Figure 2 As shown, The value of is: when When the setting is small, the accuracy of the invisible class U and the harmonic mean H tend to increase. This is because the attribute regression loss value is too small to guide the system to fully learn the attribute characteristics, and the maximum correction effect on the system is not achieved, resulting in the system's inaccurate attribute prediction. AR When the value is set to a large value, all evaluation indicators tend to decrease. This is because the loss value of the attribute regression loss is too large, which makes the system focus too much on the attribute regression task and ignore other important loss functions. At the same time, considering the accuracy of the unseen class U, the visible class S and the harmonic average H, when When the value is set to 0.003, our system achieves the most balanced performance, which shows that under this value, the attribute regression loss can properly balance the learning process of the system, neither ignoring the contribution of other loss functions, nor effectively guiding the system to learn attribute features. Moreover, under the traditional zero-shot experimental setting, Setting it to 0.003 has little effect on the experimental results. Therefore, λ in all datasets AR Set to 0.003.

[0124] About Lambda DEB The numerical setting of: DEE It is a weight used to control the effect of the debiasing loss. The appropriate value is conducive to the debiasing loss function to balance the bias problem between the visible class and the invisible class. DEB The value of is studied, such as Figure 3 As shown, λ DEB The value of is: DEB ={0.001,0.01,0.1,0.2,0.3,0.4}. When increasing λDEB When the value of , the accuracy of the unseen class increases, and the accuracy of the seen class decreases. DEE When the setting is small, the accuracy of the invisible class is significantly lower than that of the visible class. This is because the debiasing loss value is too small to effectively balance the system's bias towards the visible class. BEB When the value is set to a large value, the accuracy of the invisible class is significantly higher than that of the visible class. This is because the debiasing loss value is too large, which overemphasizes the results of the invisible class, making the accuracy results of the visible and invisible classes unbalanced. When set to 0.2, the system can also achieve the highest value in the traditional zero-shot classification setting, so the debiasing loss weight λ in all datasets DEB Set to 0.2.

[0125] Randomly select some data from the visible and invisible classes of the CUB dataset, and use the trained system to evaluate the selected data for qualitative analysis. Select the top five attributes with the highest evaluation scores for each sample, such as Figure 4 As shown in the figure, these attribute descriptions and the sample images that match them are shown. On the visible class side, each sample has at least 4 attribute descriptions that closely match the characteristics of the birds in the image. For example, in the sample "Lvory Gull", its first five attribute descriptions can all correspond to the characteristics of the birds in the sample image. On the invisible class side, each sample has at least 3 attribute descriptions that can match the description of the birds in the sample image. Therefore, the image classification method of the present invention has achieved remarkable performance in both visible and invisible class attribute recognition.

[0126] The method of the present invention is compared with other methods on three datasets: CUB, AWA2 and SUN. The results are shown in Table 1.

[0127] Table 1 Comparative experimental results on three datasets

[0128]

[0129] In the traditional zero-shot classification task, the method of the present invention achieved the best accuracy of 76.8% and 64.4% in CZSL on the CUB and SUN datasets, respectively. These results highlight the system's ability to handle different coarse and fine granularity classifications. On the dataset AWA2, the accuracy (acc) of the present invention in CZSL also showed a competitive performance of 65%. Compared with other methods that performed best on CUB and SUN, the accuracy (acc) of the present invention in CZSL achieved significant advantages of 4.8% and 0.6% over the advanced methods, respectively. It shows that the present invention provides a comprehensive, robust and generalized feature space for zero-shot classification tasks to obtain better resolution.

[0130] In the generalized zero-shot classification task, compared with the best performing method on CUB, the harmonic mean (H) of the present invention has an advantage of 0.2% over the advanced method, and the harmonic mean (H) has a more competitive result on the AWA2 dataset. In CUB, the accuracy difference between the visible class S and the invisible class U of TCN is the smallest, which is 0.6%, while the accuracy difference between the visible class S and the invisible class U of the present invention is only 0.2%, and the accuracy is far higher than that of TCN. This comparison result shows the superiority of the present invention in balancing the visible and invisible classes, and also highlights its great potential in ZSL tasks.

[0131] In AWA2, the accuracy difference between the visible class S and the invisible class U is particularly obvious, with a general difference of 15% to 20%. More extreme is that the accuracy difference between the visible class S and the invisible class U of AREN reaches 77.3%. Although the balancing effect of the present invention is not optimal on AWA2, compared with f-CLSWGAN which has achieved the best balancing effect, the method of the present invention has a higher accuracy. As a result, the image classification method of the present invention has great potential in generalized zero-shot classification tasks.

[0132] (II) Ablation experiment

[0133] In order to further understand the effectiveness of each module and feature of the system, an ablation study was conducted to evaluate the feature fusion FFT, local feature FFT in FFT, and the performance of the proposed system on the CUB dataset. l , the significant local features F in FFT max , the mixed feature F in FFT mix , attribute regression loss function And the debiasing loss function The effect is shown in Table 2.

[0134] Table 2 Ablation experiment on CUB dataset

[0135]

[0136] When the feature fusion FFT is not used, the performance of the system is significantly lower than that of the system of the present invention, and the accuracy acc and harmonic average H decrease by 14.4% and 12.7% respectively. l , salient local features F max and the mixed feature F mix The incomplete system obtains inferior results compared with the system of the present invention, where there is no local feature F lThe accuracy of the experiment dropped most significantly, with the accuracy acc and harmonic mean H dropping by 12.1% and 9.4% respectively. When using FFT without local features, the performance of the system on the dataset dropped sharply because local features help highlight those visual features that are more relevant to semantic features.

[0137] Debiasing loss can effectively alleviate the bias problem. When the debiasing loss function is removed It can be seen that the system is obviously more biased towards the visible class, and the difference between the accuracy of the visible class S and the accuracy of the invisible class U reaches 20.2%. After that, the harmonic mean H is increased by 6.7% and the influence of bias is largely eliminated. The attribute regression loss function The performance is further improved by guiding the system to perform effective visual-semantic mapping.

Claims

1. A zero-shot learning-based image classification method, characterized in that: The following steps are involved: Step 1: Building an image classification system, the system includes a feature extraction module, a feature fusion module FFT, a feature interaction module FIN and an optimization module; Step 2: Use the visible class dataset as the training set and the invisible class dataset as the test set, and obtain the visual global features and semantic features through the CNN visual backbone and GloVe; Step 3: extracting features of different dimensions from visual features and semantic features through the feature extraction module, and fusing them in the feature fusion module FFT to generate fused visual features containing multiple high dimensions; the features of different dimensions include global features, local features, mixed features and significant local features; Step 4: In the feature interaction module FIN, the fused visual features are mapped to the semantic space through a mapping function to obtain the semantic embedding of the corresponding visual features. Then the system is optimized through the optimization module; Step 5: After training is completed, first obtain the test instance x i Semantic embedding in semantic space Then use nearest neighbor matching and explicit self-calibration to predict x i Tags * ; where c * represents the predicted label, C u / C correspond to zero-shot image classification under CZSL / GZSL settings, respectively. c is the semantic embedding of category c, and II is the indicator function; By calculating the semantic embedding of the test sample and the category semantic vector z c The dot product represents the compatibility or similarity score between the sample feature and the category semantics; finally, for each test sample x i , the system will select the category c with the highest compatibility score as the predicted category.

2. The image classification method based on zero-sample learning according to claim 1, characterized in that: The process of obtaining the visual global features and semantic features described in step 2 is as follows: In generalized zero-shot learning, the training data of the visible class is used Total C seen categories, of which represents training sample i, Represents the category corresponding to sample i; there is also a data set of unseen classes represents the invisible class training sample i, Represents the category corresponding to the invisible class training sample i; there is a category c∈C seen ∪C unseen = C, whose semantic vector set is z containing attribute α c ={z1,z2,…,z α }; According to the attribute description in the semantic vector set, the language model GloVe is used to extract the attribute name f from each attribute. sem The semantic attribute information F is learned from sem ; Use CNN to extract the original feature f cv Visual feature extraction global feature F g , the formula is as follows: F g ,F sem =CNN(f cv ),GloVe(f sem )。 3. The image classification method based on zero-sample learning according to claim 2, characterized in that: The generation of the fused visual features in step 3 includes the following steps: (1) Global feature extraction: Extract the global feature F from step 2 g Methods; (2) Combining visual features with semantic features to enhance the expressive power of features and to classify unseen categories; The interaction between visual features and semantic features is calculated to generate the interaction feature α, and then the interaction feature α is normalized to generate the feature weight A, so as to convert the interaction feature into a weight for subsequent feature calculation. The formula is: where α i is the i-th element of the input interaction feature α in the last dimension, j is the index of the sum of all categories, and the sum is the sum of all elements in the last dimension; (3) Local feature extraction Use feature weight A to adjust global feature F g Weighted to generate new weighted visual features, namely new high-level local features F l , better capture the details of the image, the formula is as follows: Where r is the index of the summation process, ranging from 1 to R, and R is the size of the last dimension of the feature; (4) Hybrid feature extraction Calculate the global feature F g With local features F l The feature weight λ between them is then used to adjust the global feature F g With local features F l Perform weighted summation to generate mixed features F mix , mixed feature F mix The formula is as follows: F mix =λ·F g +(1-λ)·Fl l (5) Extraction of salient local features By calculating the global feature F g and local features F l The element-wise maximum value of max , F max The calculation formula is as follows: F max =max(F g ,F l ) (6) Feature Fusion The global feature F g , local features F l , mixed features F mix And the significant local features F max The final feature representation F is obtained by fusion, and the formula is as follows: in is the fusion coefficient.

4. The image classification method based on zero-sample learning according to claim 3, characterized in that: Semantic embedding as described in step 4 The process of obtaining is: using semantic feature F sem The semantic attribute vector in As a support vector, based on a mapping function M, the local enhanced visual feature F is combined with the semantic attribute information V α Matching, through the mapping function M, the fused visual feature F is mapped to the semantic attribute information V α Consistent semantic embedding space, the output of the feature interaction module FIN, that is, the embedding vector mapped to the semantic attribute space Where W is the embedding matrix that embeds the fusion feature F into the semantic attribute space, is the semantic attribute vector V α The transpose of .

5. The image classification method based on zero-sample learning according to claim 4, characterized in that: The optimization module described in step 4 optimizes the system by introducing the loss function To achieve this, the loss function is in is the attribute regression loss function, λ AR is the attribute regression loss weight, is the attribute cross entropy loss function, is the debiasing loss function, λ DEB is the debiasing loss weight; Attribute regression loss function Where i is the index of the summation process, ranging from 1 to n β , n β Sample size, z c is the basic real attribute, is the embedding attribute score; Attribute Cross Entropy Loss Function C s is the visible class in all class samples C, c is a real class label, is the predicted label type, represents the set of all possible categories; Debiasing loss function In the formula and σ s represents the mean and variance of the prediction scores of the visible class predictions, and σ u Represents the mean and variance of the prediction scores for the unseen classes.

Images