Zero sample learning method based on instance image attribute generation and correction

Abstract

The invention discloses a generalized zero sample learning method based on instance image attribute generation and correction, and the method comprises the steps: training a PIAS model for image classification, the PIAS model comprises an attribute generation module and an attribute correction module, and the method specifically comprises the steps: generating a class average image of each class; generating an instance attribute of the instance image and a class-level attribute of the class average image by adopting an attribute generation module; in the attribute correction module, classification loss, calibration loss and structural consistency loss are adopted to control structural alignment of the semantic space and the visual space; inputting the test data set into an attribute generation module, generating an attribute of each test image in the test data set, and inputting the attribute of each test image into a ZSL classifier to obtain a label of the test image; and determining classification precision according to the label of the test image, and judging whether the classification precision and the total loss function of the PIAS model meet preset conditions, if so, ending the algorithm, and otherwise, returning to the attribute generation module to generate attributes.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a zero-sample learning method based on instance image attribute generation and correction. Background Art

[0002] Generalized Zero-Shot Learning (GZSL) has broad application prospects in various real-world scenarios, especially in tasks where unseen categories frequently appear. GZSL aims to greatly improve the practicality and adaptability of the model by enabling it to effectively recognize categories that have not been seen during training through the generalization ability of the model. For example, in fields such as image recognition, natural language processing, and even autonomous driving, models often face inputs from unseen categories, and GZSL provides a potential solution for such scenarios. However, most current GZSL methods rely on standardized benchmark datasets with per-category attribute annotations. Although these datasets provide detailed category attribute information, they also bring new problems. Specifically, since the attribute annotations of these benchmark datasets are usually based on predefined per-category features, they may not cover all possible nuances and complex semantics, making it difficult for the model to accurately match category attributes when encountering unseen categories in practical applications. This difference leads to the so-called "semantic gap", that is, the model cannot accurately map between visual features and semantic representations, resulting in a deviation in understanding.

[0003] On the other hand, methods that rely on these attribute annotations often exacerbate the domain shift problem in the visual-semantic space. The so-called domain shift refers to the phenomenon that the performance of the model degrades when faced with different data distributions or semantic features during training and testing. In GZSL, the domain shift problem is particularly evident because the model must not only cope with changes in visual features, but also maintain stable performance in the semantic space. When there are significant differences in category attributes or features between the training data and the test data, the performance of the model often degrades severely. Therefore, although existing methods can achieve good performance in a controlled environment, in actual complex real-world scenarios, this strategy that relies on attribute annotation of benchmark datasets is insufficient and cannot effectively deal with domain shift and semantic gap problems. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the zero-shot learning method based on instance image attribute generation and correction provided by the present invention solves the problem that the existing methods cannot effectively cope with domain migration and semantic gap when applied to complex practical scenarios when attribute annotations are insufficient.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A generalized zero-shot learning method based on instance image attribute generation and correction is provided. The method is to train a PIAS model for image classification. The PIAS model includes an attribute generation module and an attribute correction module, and specifically includes the following steps:

[0007] S1. Generate the class average image of each class based on all instance images of each class in the visible class dataset;

[0008] S2, constructing a visual space using the visual transformer ViT of the attribute generation module to extract the visual features of each instance image in the visible class dataset and the average visual features of the class average image;

[0009] S3, according to the visual features of the instance image and the class average image, using the multi-layer perceptron of the attribute generation module to generate instance attributes of the instance image and class-level attributes of the class average image;

[0010] S4. In the attribute correction module, classification loss is used to make each instance attribute consistent with its corresponding category label;

[0011] S5, construct the semantic space using calibration loss to minimize the cosine similarity between the average attributes of the class and the annotation attributes;

[0012] S6, using attribute annotations and average visual features of each class as anchors for the corresponding class to control the structural alignment of semantic space and visual space;

[0013] S7, input the test data set into the attribute generation module, generate the attributes of each test image in the test data set, and input the attributes of each test image into the ZSL classifier to obtain the label of the test image;

[0014] S8. Determine the classification accuracy according to the label of the test image, and judge whether the classification accuracy and the total loss function of the PIAS model meet the preset conditions. If so, end the algorithm, otherwise return to step S2.

[0015] Furthermore, step S6 further includes:

[0016] S61. Calculate the attribute fusion matrix based on the attribute annotation of the class and the instance attributes of all images of the corresponding class:

[0017]

[0018] in, is the attribute fusion matrix of the i-th category; a i Annotate the attributes of the i-th category; is the attribute set of all instance images of the i-th category; and are the 1st and Nth images in the i-th category respectively.i N instance attributes of instance images; i is the total number of instance images of the i-th category; T is the transpose;

[0019] S62. Calculate the visual feature fusion matrix based on the average visual features of the class and the visual features of all images of the corresponding class:

[0020]

[0021] in, is the visual feature fusion matrix of the i-th category; is the average visual feature of the i-th category; and are the 1st and Nth in the i-th category respectively. i The visual features of the example images; H i is the set of visual features of all instance images of the i-th category;

[0022] S63. Control the structural alignment of the semantic space and the visual space according to the topological structure similarity between the fusion matrix of the attributes and the fusion matrix of the visual features:

[0023]

[0024]

[0025] in, is the structural consistency loss; ls is the total number of classes in the visible class dataset; for The value corresponding to the jth instance image in ; for The value corresponding to the j-th instance image in ; exp(.) is the exponential function.

[0026] Furthermore, the expression of the calibration loss is:

[0027]

[0028] in, is the calibration loss; is the class-level attribute of the i-th class; a i is the attribute annotation of the i-th class; ls is the total number of classes in the visible class dataset; cos(.) is the cosine function.

[0029] Furthermore, the classification loss is expressed as:

[0030]

[0031] in, is the classification loss; ls is the total number of classes in the visible class dataset; y i is the category label; exp(.) is the exponential function; Annotate the attributes of all classes in the visible class dataset a i A collection of; is the instance attribute of the jth instance image of the i-th category; N i is the total number of instance images of the i-th category; T is the transpose.

[0032] Furthermore, the total loss function of the PIAS model is expressed as:

[0033]

[0034] in, is the total loss; α, β, and δ are hyperparameters used to balance different loss terms; is the loss of structural consistency; is the calibration loss; is the classification loss.

[0035] Furthermore, the expression for generating the class average image is:

[0036]

[0037] in, is the class average image of the i-th class in the visible class dataset; is the jth instance image in the i-th category; N i is the total number of instance images of the i-th category.

[0038] Further, the ZSL classifier is a CZSL classifier or a GZSL classifier;

[0039] The expression of the CZSL classifier is:

[0040]

[0041] in, is the predicted label; is the attribute of the test image; y is the label; is the label of the unseen category; is the attribute of the unseen category; T is transposition; max is the maximum value;

[0042] The expression of the GZSL classifier is:

[0043]

[0044] in, is the label of the visible category; are all attributes of the seen and unseen classes; γ is the calibration factor.

[0045] Furthermore, the preset condition is that the classification accuracy is less than the preset accuracy and the change in the value of the total loss function of the PIAS model during multiple consecutive trainings is less than the preset value.

[0046] The beneficial effects of the present invention are as follows: the present solution simplifies the process of feature extraction and attribute generation through the visual transformer ViT and the multi-layer perceptron (MLP), effectively reduces the computational complexity, and maintains the high performance of the model. By correcting the visual attributes of each instance through the attribute correction module, the accuracy and diversity of the attributes can be improved, the consistency of the attributes with the real class attributes can be significantly enhanced, and the recognition rate of the unseen class can be directly improved.

[0047] This scheme can realize the synthesis of per-instance attributes with spatial information through the patch and position embedding technology introduced by the visual transformer ViT; by introducing calibration loss to map class anchors, and introducing structural consistency loss to align the topological structure between visual space and semantic space; these two losses promote diversity within the category and maintain the semantic consistency of per-instance attributes, further alleviating the semantic gap and domain shift problems in the ZSL classifier.

[0048] This scheme can ensure close consistency between generated attributes and visual features by controlling the structural alignment of semantic space and visual space, solving the domain shift problem in traditional zero-shot learning. Through this alignment, the recognition accuracy of the model in generalized zero-shot learning (GZSL) tasks is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Flowchart of a generalized zero-shot learning approach for instance-based image attribute generation and correction.

[0050] Figure 2 This is the principle block diagram of the PIAS model. DETAILED DESCRIPTION

[0051] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0052] like Figure 2As shown, this scheme designs a PIAS model, which includes an attribute generation module and an attribute correction module. The attribute generation module includes a visual transformer ViT and a multi-layer perceptron, and the attribute correction module includes three losses, namely structural consistency loss, calibration loss and classification loss.

[0053] Reference picture, Figure 1 A generalized zero-shot learning method based on instance image attribute generation and correction is shown. The method S is for training a PIAS model for image classification, and includes steps S1 to S8.

[0054] In step S1, the class average image of each class is generated based on all instance images of each class in the visible class dataset:

[0055]

[0056] in, is the class average image of the i-th class in the visible class dataset; is the jth instance image in the i-th category; N i is the total number of instance images of the i-th category.

[0057] In step S2, the visual transformer ViT of the attribute generation module is used to construct the visual space to extract the visual features of each instance image in the visible class dataset and the average visual features of the class average image:

[0058]

[0059] in, is the average visual feature of the i-th category; is the visual feature of the jth instance image in the i-th category; θ is the learnable parameter of the ViT model; d x is the dimension of visual features.

[0060] In step S3, based on the visual features of the instance image and the class average image, the multi-layer perceptron of the attribute generation module is used to generate instance attributes of the instance image and class-level attributes of the class average image:

[0061]

[0062] Where φ represents the learnable parameters of MLP, d a Represents the dimension of the attribute.

[0063] The introduction of multi-layer perceptron can provide extensive visual and attribute features for the follow-up.

[0064] In step S4, in the attribute correction module, classification loss is used to maintain consistency between each instance attribute and its corresponding category label.

[0065] When implemented, the expression of the optimal classification loss of this scheme is:

[0066]

[0067] in, is the classification loss; ls is the total number of classes in the visible class dataset; y i is the category label; exp(.) is the exponential function; Annotate the attributes of all classes in the visible class dataset a i A collection of; is the instance attribute of the jth instance image of the i-th category; N i is the total number of instance images of the i-th category; T is the transpose.

[0068] In step S5, the semantic space is constructed using calibration loss to minimize the cosine similarity between the average attribute of the class and the annotation attribute; the expression of the calibration loss is:

[0069]

[0070] in, is the calibration loss; is the class-level attribute of the i-th class; a i is the attribute annotation of the i-th class; ls is the total number of classes in the visible class dataset; cos(.) is the cosine function.

[0071] In step S6, the attribute annotations and average visual features of each class are used as anchors for the corresponding class to control the structural alignment of the semantic space and the visual space.

[0072] In one embodiment of the present invention, step S6 further comprises:

[0073] S61. Calculate the attribute fusion matrix based on the attribute annotation of the class and the instance attributes of all images of the corresponding class:

[0074]

[0075] in, is the attribute fusion matrix of the i-th category; a i Annotate the attributes of the i-th category; is the attribute set of all instance images of the i-th category; and are the 1st and Nth images in the i-th category respectively. i N instance attributes of instance images; iis the total number of instance images of the i-th category; T is the transpose;

[0076] S62. Calculate the visual feature fusion matrix based on the average visual features of the class and the visual features of all images of the corresponding class:

[0077]

[0078] in, is the visual feature fusion matrix of the i-th category; is the average visual feature of the i-th category; and are the 1st and Nth in the i-th category respectively. i The visual features of the example images; H i is the set of visual features of all instance images of the i-th category;

[0079] S63. Control the structural alignment of the semantic space and the visual space according to the topological structure similarity between the fusion matrix of the attributes and the fusion matrix of the visual features:

[0080]

[0081] in, is the structural consistency loss; ls is the total number of classes in the visible class dataset; for The value corresponding to the jth instance image in ; for The value corresponding to the j-th instance image in ; exp(.) is the exponential function.

[0082] This scheme can make the generated per-instance attributes more More detailed and accurate, thus building a more refined semantic space.

[0083] In step S7, the test data set is input into the attribute generation module to generate the attributes of each test image in the test data set, and the attributes of each test image are input into the ZSL classifier to obtain the label of the test image.

[0084] When implemented, the preferred ZSL classifier in this scheme is a CZSL classifier or a GZSL classifier;

[0085] The expression of CZSL classifier is:

[0086]

[0087] in, is the predicted label; is the attribute of the test image; y is the label; is the label of the unseen category; is the attribute of the unseen category; T is transposition; max is the maximum value;

[0088] The expression of the GZSL classifier is:

[0089]

[0090] in, is the label of the visible category; are all attributes of the seen and unseen classes; γ is the calibration factor.

[0091] In step S8, the classification accuracy is determined according to the label of the test image, and it is judged whether the classification accuracy and the total loss function of the PIAS model meet the preset conditions. If so, the algorithm is terminated, otherwise it returns to step S2.

[0092] Among them, the expression of the total loss function of the PIAS model is:

[0093]

[0094] in, is the total loss; α, β, and δ are hyperparameters used to balance different loss terms; is the loss of structural consistency; is the calibration loss; is the classification loss.

[0095] The preferred preset conditions in this solution are that the classification accuracy is less than the preset accuracy and the change in the value of the total loss function of the PIAS model during multiple consecutive trainings is less than the preset value.

[0096] The effect of the zero-sample learning method proposed in this solution is described below in conjunction with specific embodiments:

[0097] This example selects three recognized challenging benchmark datasets in the field of generalized zero-shot learning (GZSL) for extensive experiments, including AWA2, CUB, and SUN. These datasets cover different fields from animals to birds to scenes, and show obvious differences in fine-grained and coarse-grained, so the performance of the PLAS model in different tasks and scenes can be comprehensively evaluated.

[0098] AWA2 (Animals with Attributes 2) is a coarse-grained dataset that focuses on the recognition of animal categories. The dataset contains 50 animal categories, divided into 40 seen categories and 10 unseen categories, with a total of 37,322 images collected. AWA2 uses 85 attributes to describe the characteristics of these animal categories, which cover important biological characteristics such as the appearance and behavior of animals. For example, attributes may include "hoofed", "herbivorous" or "furry". Since the categories in AWA2 are relatively macroscopic, each category in the dataset may contain wide differences across species, so AWA2 is considered a coarse-grained dataset. In zero-shot learning, the challenge of coarse-grained datasets is how to make the model generalize to animal categories that have never been seen through a small number of attribute features.

[0099] CUB (Caltech-UCSDBirds-200-2011) is a fine-grained dataset dedicated to bird recognition. The CUB dataset contains 11,788 images of 200 bird species, of which 150 are seen categories and 50 are unseen categories. Each bird species is described by 312 detailed attributes, which cover various characteristics of birds, such as color, morphology, feather structure, beak shape, etc. This makes CUB a typical fine-grained dataset because it requires the model to capture subtle differences between different species. Since many birds are very similar in appearance, such as similar color or body shape, the difficulty of the CUB dataset lies in how to distinguish these categories through extremely detailed attributes. Therefore, fine-grained datasets place higher requirements on the model, especially in dealing with subtle differences and complex visual features.

[0100] SUN (SUN Attribute) is a fine-grained dataset for scene recognition. The SUN dataset contains 717 scene categories and a total of 14,340 images, of which 645 are seen categories and 72 are unseen categories. Unlike animals or birds, the SUN dataset focuses on the classification of scene categories, such as urban landscapes, forests, beaches, classrooms, etc. Each scene is described by 102 attributes, which cover different visual and semantic features of the scene, such as "nature", "buildings", "water bodies", etc. Although the categories in SUN are relatively rich, the attribute features of each scene may be relatively similar. For example, many natural scenes may contain "trees" and "water bodies", which increases the complexity of the zero-shot learning task. Due to the diversity of scene recognition and the high overlap between attributes, the model needs to find key differences in complex visual information to identify unseen scene categories, which makes SUN a highly challenging fine-grained dataset.

[0101] This example uses the pre-trained ViT-Large model as the visual feature extractor. To optimize the model, the Adam optimizer is used with an initial learning rate of 10 -6 , the attenuation factor is 10 -5 . In addition, a multi-step learning rate scheduler was used to adjust the learning rate at predefined training rounds (10, 20, 30, 40 rounds) with a scaling factor of 0.5 (gamma = 0.5). This method is implemented on the PyTorch platform, using its parallel distributed training framework and trained on two NVIDIA GeForceRTX4090 GPUs. The entire training process lasted for 80 rounds with a batch size of 30 (15 images per GPU). The calibration factor γ was set to 0.9, 0.2, and 0.1 on the AWA2, CUB, and SUN datasets, respectively.

[0102] To ensure a fair comparison with previous related methods, this embodiment adopts a unified evaluation protocol. For zero-shot learning (ZSL), the top-1 (T1) average category accuracy of all unseen categories is evaluated according to the unified evaluation protocol. For generalized zero-shot learning (GZSL), the average category accuracy of seen categories (S) and unseen categories (U) is first calculated separately, and then their harmonic mean (H) is used to evaluate the performance of the method, where H = (2*S*U) / (S+U). A higher harmonic mean indicates that the method performs well on both seen and unseen categories, reflecting a balanced and robust performance.

[0103] This example compares the proposed PIAS model with a number of recent methods that have emerged in the past four years, including both generative and non-generative methods. These methods include generative methods such as HVSA 1 ,SDGZSL 2 AGZSL 3 、SE-GZSL 4 , SRSA 5 、ICCE 6 SC-EGG 7 VS-Boost 8 , EGANS 9 and CSDR 10 , and non-generative methods including ViT-ZSL 11 、Transzero 12 、Transzero++ 13 、MSDN 14 LAPE 15 , HRT 16 、PSVMA 17 , DUET 18 IEAM-ZSL19 、AS-ZSL 20 and ZSLViT 21 The 21 models in this embodiment correspond to the following references (1) to (21) respectively. The results of the PIAS model compared with the 21 models in the prior art are specifically compared with Table 1.

[0104] Table 1. Comparison of PIAS model with state-of-the-art methods under AWA2, CUB and SUN settings

[0105]

[0106]

[0107] As shown in Table 1, the proposed PIAS model achieves the best results under the GZSL setting on all datasets, measured by the harmonic mean H, 79.7% for the AWA2 dataset, 75.7% for CUB, and 63.1% for SUN. Specifically, on the AWA2 dataset, most methods show significant differences between seen and unseen categories. For example, ViT-ZSL achieves 90% accuracy on seen categories, but only 51.9% on unseen categories. The imbalance in accuracy between seen and unseen categories is a common problem caused by domain transfer. Therefore, although these methods perform well on seen categories, their overall harmonic mean performance is still not ideal due to their poor performance on unseen categories.

[0108] Compared with the above methods, although the PIAS model proposed in this scheme did not achieve the highest performance on both seen categories (87.9%) and unseen categories (72.9%), it achieved a good balance between the two. This balance made the harmonic mean performance of this dataset reach the best (79.7%). This improvement is attributed to the structural alignment module introduced in this scheme, which effectively solves the problem of domain transfer.

[0109] On the fine-grained datasets CUB and SUN, the proposed method achieves state-of-the-art performance on all metrics, namely S, U, and H. Similar to the second-best method LAPE, PIAS enhances semantic attribute features from pixels. However, PIAS outperforms LAPE on the CUB dataset, showing a balanced improvement of about 1%, indicating its superior ability in bridging the visual-semantic gap. Since the SUN dataset has low inter-class similarity and high class diversity, knowledge transfer and feature capture face significant challenges, making it one of the datasets with the greatest potential for improvement in zero-shot learning. For the first time, our method achieves an accuracy of over 60% on the SUN dataset, with a harmonic mean of 63.1%, which is a 10.8% improvement over the previous best method PSVMA. Overall, in GZSL, PIAS demonstrates excellent comprehensive capabilities and successfully copes with the challenges of all datasets. These results not only highlight the superiority of PIAS, but also emphasize its consistency and robustness across different datasets.

[0110] This example also shows the comparison results under the traditional ZSL setting on the AWA2, CUB and SUN datasets. It is worth emphasizing that the method of this scheme performs best on the SUN dataset with an accuracy of 76.4%, which is 8.8% higher than the previous best method TransZero++. SUN is a scene dataset, and generative methods are generally able to adapt to more features and therefore achieve better results on unseen categories. However, the PIAS model excels in understanding the abstract content of the scene by mapping visual features to semantic space to synthesize the attributes of each instance. The results significantly surpass non-generative and generative methods, further confirming the effectiveness of the method of this scheme in synthesizing the attributes of each instance and demonstrating excellent generalization capabilities.

[0111] On the AWA2 dataset, PIAS surpasses all non-generative methods and most generative methods, only 0.1% lower than the second best result (76.4%). This shows that our method performs well on coarse-grained datasets. On the CUB dataset, our method fails to achieve the best performance under the CZSL setting. Analysis shows that this may be due to the ultra-fine-grained nature of the CUB dataset, whose attribute feature dimension reaches 312, three times that of the fine-grained SUN dataset. For non-generative methods, small changes in attribute features pose significant challenges and seriously affect the accuracy of unseen categories.

[0112] In summary, the PIAS model of this scheme effectively addresses the key technical challenges in zero-shot learning, namely the management of domain shift and attribute dimensions, through its structural alignment module and optimized processing of fine-grained features, thereby significantly improving the accuracy and generalization ability of the learning model.

[0113] References (1) to (21):

[0114] (1) Chen, S.; Xie, G.; Liu, Y.; Peng, Q.; Sun, B.; Li, H.; You, X.; Shao, L. HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning. In Advances in Neural Information Processing Systems; Curran Associates, Inc., 2021; Vol. 34, pp16622–16634.

[0115] (2) Chen, Z.; Luo, Y.; Qiu, R.; Wang, S.; Huang, Z.; Li, J.; Zhang, Z. Semantics Disentangling for Generalized Zero-Shot Learning. In 2021 IEEE / CVF International Conference on Computer Vision (ICCV); IEEE: Montreal, QC, Canada, 2021; pp8692–8700. https: / / doi.org / 10.1109 / ICCV48922.2021.00859.

[0116] (3) Chou, Y.-Y.; Lin, H.-T.; Liu, T.-L. ADAPTIVE AND GENERATIVE ZERO-SHOT LEARNING. 2021.

[0117] (4) Kim, J.; Shim, K.; Shim, B. Semantic Feature Extraction for Generalized Zero-Shot Learning. Proc. AAAI Conf. Artif. Intell. 2022, 36(1), 1166–1173. https: / / doi.org / 10.1609 / aaai.v36i1.20002.

[0118] (5)Liu,Y.;Gao,X.;Han,J.;Liu,L.;Shao,L.Zero-ShotLearningviaSpecificRank-ControlledSema nticAutoencoder.PatternRecognit.2022,122,108237.https: / / doi.org / 10.1016 / j.patcog.2021.108237.

[0119] (6)Kong,X.; Gao,Z.;Li, enceonComputerVisionandPatternRecognition(CVPR);IEEE:NewOrleans,LA,USA,2022;pp9296–9305.https: / / doi.org / 10.1109 / CVPR52688.2022.00909.

[0120] (7)Hong,Z.;Chen,S.;Xie,G.-S.;Yang,W.;Zhao,J.;Shao,Y.;Peng,Q.;You,X.SemanticCompressio nEmbeddingforGenerativeZero-ShotLearning.InProceedingsoftheThirty-FirstInternationalJointCon ferenceonArtificialIntelligence,IJCAI-22;Raedt,LD,Ed.;InternationalJointConferencesonArtificialIntelligenceOrganization,2022;pp956–963.https: / / doi.org / 10.24963 / ijcai.2022 / 134.

[0121] (8)Li,X.;Zhang,Y.;Bian,S.;Qu,Y.;Xie,Y.;Shi,Z.;Fan,J.VS-Boost:BoostingVisual-SemanticAsso ciationforGeneralizedZero-ShotLearning.InProceedingsoftheThirty-SecondInternationalJointConfer enceonArtificialIntelligence;InternationalJointConferencesonArtificialIntelligenceOrganization:Mac au,SARChina,2023;pp1107–1115.https: / / doi.org / 10.24963 / ijcai.2023 / 123.

[0122] (9)Chen,S.;Chen,S.;Hou,W.;Ding,W.;You,X.EGANS:EvolutionaryGenerativeAdversarialNet workSearchforZero-ShotLearning.arXivAugust19,2023.http: / / arxiv.org / abs / 2308.09915(accessed2023-12-05).

[0123] (10)Gao,Y.;Feng,W.;Xiao,R.;He,L.;He,Z.;Lv,J.;Tang,C.ImprovingGeneralizedZero-ShotLearn ingviaCluster-BasedSemanticDisentanglingRepresentation.PatternRecognit.2024,150,110320.https: / / doi.org / 10.1016 / j.patcog.2024.110320.

[0124] (11)Alamri,F.;Dutta,A.Multi-HeadSelf-AttentionviaVisionTransformerforZero-ShotLearning.arXivJuly30,2021.http: / / arxiv.org / abs / 2108.00045(accessed2023-11-13).

[0125] (12)Chen,S.Hong,Z.Liu,Y.Xie,G.-S.Sun,B.Li,H.Peng,Q.Lu,K.You,X.TransZero:Attribute e-GuidedTransformsforZero-ShotLearning.Proc.AAAIConf.Artif.Intell.2022,36(1),330–338.

[0126] (13)Chen,S.Hong,Z.Hou,W.Xie,G.-S.Song,Y.Zhao,J.You,X.Yan,S.Shao,L.TransZero++:CrossAttribute-GuidedTransf wormsforZero-ShotLearning.IEEETrans.PatternAnal.Mach.Intell.2023.1–17.https: / / doi.org / 10.1109 / TPAMI.2022.3229526.

[0127] (14)Chen,S.Hong,Z.Xie,G.-S.Yang,W.Peng,Q.Wang,K.Zhao,J.You,X.MSDN:MutuallySemanticDistillationNetworkforZero-Sh otLearning.In2022IEEE / CVFConferenceonComputerVisionandPatternRecognition(CVPR)IEEE:NewOrleans,LA,USA,2022:pp7602–7611.

[0128] https: / / doi.org / 10.1109 / CVPR52688.2022.00746.

[0129] (15)Wang,Z.;Gou,Y.;Li,J.;Zhu,L.;Shen,H.T.Language-AugmentedPixelEmbeddingforGeneralizedZero-ShotLearning.IEEETrans.CircuitsSyst.VideoTechnol.2023,33(3),1019–1030.

[0130] https: / / doi.org / 10.1109 / TCSVT.2022.3208256.

[0131] (16)Cheng,D.;Wang,G.;Wang,B.;Zhang,Q.;Han,J.;Zhang,D.HybridRoutingTransformerforZero-ShotLearning.PatternRecognit.2023,137,109270.https: / / doi.org / 10.1016 / j.patcog.2022.109270.

[0132] (17)Liu,M.;Li,F.;Zhang,C.;Wei,Y.;Bai,H.;Zhao,Y.ProgressiveSemantic-VisualMutualAdaptionforGeneralizedZero-ShotLearning.In2023IEEE / CVFConferenceonComputerVisionandPatternRecognition(CVPR);IEEE:Vancouver,BC,Canada,2023;pp15337–15346.

[0133] https: / / doi.org / 10.1109 / CVPR52729.2023.01472.

[0134] (18)Chen,Z.;Huang,Y.;Chen,J.;Geng,Y.;Zhang,W.;Fang,Y.;Pan,JZ;Chen,H.DUET:Cross-ModalSemanticGroundingforContra stiveZero-ShotLearning.Proc.AAAIConf.Artif.Intell.2023,37(1),405–413.https: / / doi.org / 10.1609 / aaai.v37i1.25114.

[0135] (19)Alamri,F.;Dutta,A.ImplicitandExplicitAttentionMechanismsforZero-ShotLearning.Neurocomputing2023,534,55–66.https: / / doi.org / 10.1016 / j.neucom.2023.03.009.

[0136] (20)Zhou,L.;Liu,Y.;Bai,X.;Li,N.;Yu,X.;Zhou,J.;Hancock,ERAttributeSubspacesforZero-Shot Learning.PatternRecognit.2023,144,109869.https: / / doi.org / 10.1016 / j.patcog.2023.109869.

[0137] (21)Chen,S.;Hou,W.;Khan,S.;Khan,FSProgressiveSemantic-GuidedVisionTransformerforZero-ShotLearning.arXivApril11,2024.https: / / doi.org / 10.48550 / arXiv.2404.07713。

Claims

1. A generalized zero-shot learning method based on instance image attribute generation and correction, characterized in that: The method is to train a PIAS model for image classification, wherein the PIAS model includes an attribute generation module and an attribute correction module, and specifically includes the following steps: S1. Generate the class average image of each class based on all instance images of each class in the visible class dataset; S2, constructing a visual space using the visual transformer ViT of the attribute generation module to extract the visual features of each instance image in the visible class dataset and the average visual features of the class average image; S3, according to the visual features of the instance image and the class average image, using the multi-layer perceptron of the attribute generation module to generate instance attributes of the instance image and class-level attributes of the class average image; S4. In the attribute correction module, classification loss is used to make each instance attribute consistent with its corresponding category label; S5, construct the semantic space using calibration loss to minimize the cosine similarity between the average attributes of the class and the annotation attributes; S6, using attribute annotations and average visual features of each class as anchors for the corresponding class to control the structural alignment of semantic space and visual space; S7, input the test data set into the attribute generation module, generate the attributes of each test image in the test data set, and input the attributes of each test image into the ZSL classifier to obtain the label of the test image; S8. Determine the classification accuracy according to the label of the test image, and judge whether the classification accuracy and the total loss function of the PIAS model meet the preset conditions. If so, end the algorithm, otherwise return to step S2.

2. The zero-shot learning method based on instance image attribute generation and correction according to claim 1, characterized in that: Step S6 further comprises: S61. Calculate the attribute fusion matrix based on the attribute annotation of the class and the instance attributes of all images of the corresponding class: in, is the attribute fusion matrix of the i-th category; a i Annotate the attributes of the i-th category; is the attribute set of all instance images of the i-th category; and are the 1st and Nth images in the i-th category respectively. i N instance attributes of instance images; i is the total number of instance images of the i-th category; T is the transpose; S62. Calculate the visual feature fusion matrix based on the average visual features of the class and the visual features of all images of the corresponding class: in, is the visual feature fusion matrix of the i-th category; is the average visual feature of the i-th category; and are the 1st and Nth in the i-th category respectively. i The visual features of the example images; H i is the set of visual features of all instance images of the i-th category; S63. Control the structural alignment of the semantic space and the visual space according to the topological structure similarity between the fusion matrix of the attributes and the fusion matrix of the visual features: in, is the structural consistency loss; ls is the total number of classes in the visible class dataset; for The value corresponding to the jth instance image in ; for The value corresponding to the j-th instance image in ; exp(.) is the exponential function.

3. The zero-shot learning method based on instance image attribute generation and correction according to claim 1, characterized in that: The expression of the calibration loss is: in, is the calibration loss; is the class-level attribute of the i-th class; a i is the attribute annotation of the i-th class; ls is the total number of classes in the visible class dataset; cos(.) is the cosine function.

4. The zero-shot learning method based on instance image attribute generation and correction according to claim 1, characterized in that: The expression of the classification loss is: in, is the classification loss; ls is the total number of classes in the visible class dataset; y i is the category label of the i-th category; exp(.) is the exponential function; Annotate the attributes of all classes in the visible class dataset a i A collection of; is the instance attribute of the jth instance image of the i-th category; N i is the total number of instance images of the i-th category; T is the transpose.

5. The zero-shot learning method based on instance image attribute generation and correction according to any one of claims 1 to 4, characterized in that: The expression of the total loss function of the PIAS model is: in, is the total loss; α, β, and δ are hyperparameters used to balance different loss terms; is the loss of structural consistency; is the calibration loss; is the classification loss.

6. The zero-shot learning method based on instance image attribute generation and correction according to claim 1, characterized in that: The expression for generating the class average image is: in, is the class average image of the i-th class in the visible class dataset; is the jth instance image in the i-th category; N i is the total number of instance images of the i-th category.

7. The zero-shot learning method based on instance image attribute generation and correction according to claim 1, characterized in that: The ZSL classifier is a CZSL classifier or a GZSL classifier; The expression of the CZSL classifier is: in, is the predicted label; is the attribute of the test image; y is the label; is the label of the unseen category; is the attribute of the unseen category; T is transposition; max is the maximum value; The expression of the GZSL classifier is: in, is the label of the visible category; are all attributes of the seen and unseen classes; γ is the calibration factor.

8. The zero-shot learning method based on instance image attribute generation and correction according to claim 1, characterized in that: The preset conditions are that the classification accuracy is less than the preset accuracy and the change in the value of the total loss function of the PIAS model during multiple consecutive trainings is less than the preset value.

Images