Zero sample image classification method and device

Abstract

The invention discloses a zero sample image classification method and device, and belongs to the field of image recognition. According to the method, a category description document is generated through a large language model, global and local feature information of an image and the document is obtained through a visual encoder and a text encoder, then decoupling is performed through a visual semantic decomposition module and a text semantic decomposition module, and visual semantic embedding and text semantic embedding are generated; through joint training of the modules, optimizing total loss composed of local-to-semantic mapping variance loss, multi-semantic diversity loss, decomposition semantic alignment loss and local semantic alignment loss; and the training is used for image category prediction in the reasoning stage after completion of training. According to the invention, the difficulty of zero sample image classification can be reduced, and the classification accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition, and in particular relates to a zero-sample image classification method and device. Background Art

[0002] Zero-shot image classification is an important technology in the field of computer vision and machine learning. Its purpose is to accurately identify unseen categories without relying on new training images. Traditional zero-shot image classification methods [1-4] usually rely on manually annotated attributes as auxiliary information to enable classifiers to be trained on known categories and further applied to the recognition of unknown categories. Specifically, these methods map visual and text embeddings into a common space by utilizing compatibility functions; or combine generative models to generate samples of unseen categories, transforming the zero-shot image classification task into a traditional supervised task; and promote knowledge transfer by enhancing the visual semantic embedding between attributes and local image regions. However, such methods have obvious disadvantages, namely that the process of annotating attributes requires a lot of human resources and in-depth domain expertise, and as new categories on the network increase, the need for recognition will require more attribute annotations, which undoubtedly increases the difficulty of implementation.

[0003] Compared with traditional methods, unsupervised semantic embedding provides a new solution that can utilize auxiliary information without human intervention. Specific implementation methods [5-7] include utilizing word embeddings in pre-trained language models, using the relationship between category embeddings in a common semantic space, and aligning word vector space with visual space to achieve knowledge transfer. Some advanced methods integrate knowledge graphs to establish the relationship between categories and category word vectors, further enhancing the semantic connection and thus optimizing classification performance. However, since these methods often lack discriminative visual information and are highly sensitive to language issues, their performance in practical applications is relatively limited.

[0004] At the same time, with the rapid growth of network information, a large amount of document data has become an important knowledge resource. These documents, such as Wikipedia, contain rich human domain knowledge and fine-grained information, providing new ideas for zero-shot image classification. Specific implementation plans include: obtaining semantic information related to the target category by directly using the document embedding of TF-IDF or large language models as auxiliary information [8-9]. Secondly, some advanced methods enhance the embedding through partial detection networks and further use annotation attributes to train the model to achieve more accurate classification

[10] . Recently, some works have optimized classification performance by learning fine-grained interactions to enhance semantic embedding [11-12]. For example, I2DFromer

[11] trained a transformer-based model that can align the global and local compatibility functions of image patches and words, thereby achieving more accurate classification. In addition, I2MVFormer

[12] aggregates information at the document level and aligns it with image regions, thereby reducing the computational cost during training. Moreover, I2MVFormer also uses large language models to generate multi-view category documents, providing richer semantic information for the recognition of unseen categories.

[0005] The current zero-shot image classification technology has the following problems:

[0006] 1. Zero-shot image classification technology using attribute annotation requires more manual annotation and in-depth expert knowledge in the field, and with the increase of new categories, more and more attribute annotation information is required; while the unsupervised method based on category word vectors lacks visual information content and is highly sensitive to language, resulting in lower performance.

[0007] 2. The zero-shot image classification technology based on documents ignores some of the associations between images and texts: images only reflect part of the information in documents, and different images reflect different document information. This leads to suboptimal alignment between document space and image space, which damages the ability to transfer knowledge from visible categories to unseen categories and undermines the performance of the model.

[0008] 3. The means of obtaining documents using zero-shot image classification technology are relatively simple: obtaining them from Wikipedia, or using large language models to rewrite Wikipedia descriptions, resulting in a lack of visual semantic information for some categories with fewer Wikipedia descriptions. Summary of the invention

[0009] The purpose of the present invention is to propose a zero-shot image classification method and device based on semantic decomposition and local correlation information alignment to solve the problems existing in the current zero-shot image classification technology.

[0010] The technical solution adopted by the present invention to achieve the above-mentioned purpose is as follows:

[0011] A zero-shot image classification method comprises the following steps:

[0012] 1) Collect web documents and use large language models to generate category description documents:

[0013] 2) The input image is passed through a visual encoder to generate global and local image features, which are then input into a visual semantic decomposition module for decoupling to generate a visual semantic embedding; at the same time, the category description document corresponding to the image is passed through a text perceptron to generate global and local text features, which are then input into a text semantic decomposition module for decoupling to generate a text semantic embedding; the information decoupling between the visual semantic embedding and the text semantic embedding is enhanced through the local-to-semantic mapping variance loss and the multi-semantic diversity loss;

[0014] 3) Establish the correlation between visual semantic embedding and text semantic embedding by decomposing semantic alignment loss, and measure the semantic similarity between images and documents using smooth chamfer similarity; align the semantic information between image patches and document word tags by local semantic alignment loss;

[0015] 4) constructing a zero-shot image classification model by using an image sensor, a text sensor, a visual semantic decomposition module, and a text semantic decomposition module, and training the model to optimize the total loss consisting of local-to-semantic mapping variance loss, multi-semantic diversity loss, decomposition semantic alignment loss, and local semantic alignment loss;

[0016] 5) In the inference stage, the trained zero-shot image classification model is used to calculate the semantic similarity between the input image and all possible categories, and the category with the highest score is selected as the category prediction result.

[0017] Furthermore, the step of using a large language model to generate a category description document in step 1) includes:

[0018] From collecting web documents related to the category;

[0019] For categories whose descriptions in documents lack detailed visual information, a large language model is used to generate supplementary visual descriptions through hint templates;

[0020] The generated supplementary visual descriptions are combined with the collected web documents to obtain category description documents.

[0021] Furthermore, in step 2), the image sensor includes a visual encoder and a residual fully connected layer; the step of generating global and local image features through the visual encoder includes: the visual encoder performs feature encoding on the input image, and the feature encoding is processed through the residual fully connected layer to generate global image features and local image features.

[0022] Furthermore, in step 2), the text perceptron includes a fully connected layer and a text encoder, and the text encoder includes two layers of transformers; the step of generating global and local text features of the document through the text perceptron includes: the text perceptron first initializes the words in the document with GloVe vectors, then reduces the dimension through the fully connected layer processing, and then inputs it into the text encoder to generate global text features and local text features.

[0023] Furthermore, the visual semantic decomposition module in step 2) is decoupled, and the steps of generating visual semantic embedding include:

[0024] In the initial iteration, a set of learnable tokens is input as a set embedding;

[0025] In each iteration, the collective embedding and local image features from the previous iteration are input into the aggregation block;

[0026] Map local image features to image keys k and values ​​v, and map the set embedding to the query q;

[0027] Calculate the attention map through the cross-attention mechanism;

[0028] Multiply the attention map with the image value v, and then generate a new visual semantic embedding through a learnable linear layer and residual connection;

[0029] After multiple iterations, the final visual semantic embedding is obtained.

[0030] Furthermore, the text semantic decomposition module in step 2) is decoupled, and the corresponding steps of generating text semantic embedding include:

[0031] In the initial iteration, a set of learnable tokens is input as a set embedding;

[0032] In each iteration, the collective embedding and local text features of the previous iteration are input into the aggregation block;

[0033] Map local text features to text keys k and values ​​v, and map set embeddings to queries q:

[0034] Calculate the attention map through the cross-attention mechanism;

[0035] Multiply the attention map with the text value v, and then generate a new text semantic embedding through a learnable linear layer and residual connection;

[0036] After multiple iterations, the final text semantic embedding is generated.

[0037] Furthermore, the step of calculating the smooth chamfer similarity in step 3) includes:

[0038] Compute the similarity between each visual semantic embedding and all textual semantic embeddings;

[0039] For each visual semantic embedding, select the P pairs of embeddings that are most similar to the textual semantic embedding; for each textual semantic embedding, select the P pairs of embeddings that are most similar to the visual semantic embedding; form a set of P×P pairs of visual-textual embedding pairs;

[0040] The smooth chamfer similarity function is used to calculate the similarity of the above visual-text embedding pairs to obtain the smooth chamfer similarity between the image and the document.

[0041] Furthermore, the step of calculating the local semantic alignment loss in step 3) includes:

[0042] Pass the local image features to a linear layer to generate a local image query q;

[0043] Pass the local text features to two linear layers to generate local text keys k and local text values ​​v respectively;

[0044] Use the local image query q and the local text key k to calculate the similarity between the two and generate an attention mapping matrix;

[0045] Perform a global pooling operation on the attention mapping matrix and aggregate it to obtain the representation H;

[0046] A linear layer is used to map the pooled representation H into a fine-grained similarity score, which is used to measure the similarity between the image and the document at the local level.

[0047] A cross entropy loss function is used to optimize the fine-grained similarity score, resulting in a local semantic alignment loss.

[0048] A zero-sample image classification device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0049] The present invention has the following advantages:

[0050] 1. The present invention generates visual document descriptions including appearance, shape, color, habitat, etc. by introducing a large language model, which greatly enriches the semantic information in categories with fewer descriptions and improves the performance of the classification model.

[0051] 2. The present invention decomposes the semantic concepts of a document into multiple representations and aligns some of the associations based on their similarity, thereby significantly improving the semantic matching between the document and the visual image and promoting the knowledge transfer capability.

[0052] 3. The present invention makes the recognition of unseen categories (invisible categories) simpler and more efficient by accurately modeling the partial association relationship between documents and images, laying a solid foundation for the development of zero-sample image classification technology in the future.

[0053] 4. Compared with the traditional zero-shot image classification method based on attribute and category word vectors, the present invention uses documents as auxiliary information for knowledge transfer, making the acquisition process simpler. Users can obtain the required information only through Wikipedia or large language models, without the need for detailed annotation by experts, thus reducing labor costs.

[0054] 5. The present invention collects relevant document information through the network, which contains a large amount of rich semantic knowledge, so that the model can use more comprehensive background information during training, thereby enhancing the robustness of the model.

[0055] 6. The present invention greatly promotes the alignment of documents and images in a common semantic space by modeling partial associations between them, and improves the knowledge transfer capability from visible categories to invisible categories.

[0056] 7. The present invention uses variance loss to make different semantic representations focus on different detail information, and uses diversity loss to make each semantic representation orthogonal to each other, further enhancing the semantic decoupling and alignment of documents and images.

[0057] 8. The present invention establishes partial associations between documents and images at the semantic level through global semantic alignment loss, and establishes fine-grained associations between image blocks and words through local alignment loss, thereby improving the effect and accuracy of knowledge transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flowchart of zero-sample image classification in an embodiment.

[0059] Figure 2 Schematic diagram of multi-semantic diversity loss and local-to-semantic mapping variance loss.

[0060] Figure 3 It is a module graph related to local semantic alignment loss.

[0061] Figure 4 It is a schematic diagram of local semantic alignment loss and decomposed semantic loss. DETAILED DESCRIPTION

[0062] In order to make the various technical features and advantages or technical effects in the above technical solutions of the present invention more obvious and easy to understand, they are described in detail below with reference to the accompanying drawings.

[0063] The embodiment of the present invention specifically proposes a method and device for zero-shot image classification based on semantic decomposition and partial alignment (Embedding Decomposition and Partial Alignment, referred to as EmDepart), which uses documents to describe each category, trains the model with seen category data, aligns the document and image space, completes the knowledge transfer from known categories to unknown categories, and thus helps identify unknown categories.

[0064] The technical idea of ​​the present invention is: first, a prompt template is proposed to enable ChatGPT to generate a visual description of the category to be classified. Subsequently, a semantic decomposition module is used to generate multiple semantic representations of information decoupled from each other for documents and pictures, which provides a semantic concept basis for the alignment of document and picture space. Finally, the decomposed semantic alignment loss and the local semantic alignment loss are used to complete the partial semantic alignment of documents and images at the semantic level and the word-to-image area level, respectively, to help align the document space and the visual space, thereby promoting the transfer of knowledge from known categories to unknown categories. After completing the model training, in the specific inference stage, the semantic similarity between the current image and all possible categories is obtained, and the one with the highest score is selected as the category prediction result.

[0065] The overall process of zero-shot image classification is as follows Figure 1 As shown, there are four core steps, including generating category description documents with the help of ChatGPT, decomposing document and image semantic representations, partial alignment of document and image space, model training and inference recognition of new category images, which are described in detail below.

[0066] 1. Generate category visual description documents with ChatGPT

[0067] The text corpus serves as auxiliary information for document-based zero-shot learning (ZSL), realizing knowledge transfer from seen categories to unseen categories. The present invention collects documents from various sources on the Internet, such as Wikipedia. Since documents may have potential noise for zero-shot image classification tasks, the present invention selects relevant chapters on categories in the encyclopedia, including but not limited to appearance, habitat, and behavior. However, some fine-grained categories are insufficiently described on the Internet, for example, the dog categories Chihuahua and Border Collie, there are very few descriptions of these categories in Wikipedia, and there are almost no descriptions related to vision. Therefore, the present invention uses a large language model ChatGPT to generate category definitions through prompt engineering to supplement the text corpus. The present invention proposes the following text generation prompt examples:

[0068] "I would like you to serve as an expert on {type} and provide visual definitions for various {types}. I will provide you with the name of the {type} and you will need to provide a detailed definition of it, preferably including visual information such as its shape, color, appearance, habitat, etc. Now, I would like you to define {class name}. Please describe it in 500 words or less."

[0069] In the above example text generation prompt, the present invention uses the coarse-grained category to be defined as {type}, such as animals, birds, and flowers. Then, {class name}, that is, the name of the category to be defined, is selected as the category for generating definitions using a large language model. The present invention combines the definition generated by the large language model and the documents collected by Wikipedia as the final auxiliary information to help transfer knowledge from seen categories to unseen categories.

[0070] 2. Decomposition of document and image semantic representations

[0071] After collecting the visual description of the category to be identified, the present invention decouples the semantic information of the image and the document to generate multiple semantic representations focusing on different information.

[0072] First, if Figure 1 As shown in the left area of ​​, the image and document will pass through the image sensor and text sensor respectively to extract features that are helpful for the zero-shot image classification task. Specifically, the image sensor includes a visual encoder (Vision Transformer, abbreviated as ViT) and a residual fully connected layer. Given an input image x, the image sensor first encodes the features through a pre-trained visual encoder. Next, a learnable fully connected layer extracts visual information that is crucial for the zero-shot image classification task, and its dimension is r. The present invention adds a residual connection to the fully connected layer to obtain general visual knowledge in ViT. The image sensor outputs the classification head ([CLS]) label as the global image feature l CLS (x)∈R r , and output local image features, (x)∈R N×r , where N is the number of image patches. On the text side, the text perceiver contains a fully connected layer and a text encoder. Given a document d containing M words, the present invention uses GloV e The vector assigns input features to each word. The text perceptron first passes these features through a learnable fully connected layer of dimension r to reduce computational cost, resulting in a text sequence feature. The present invention then adds a [CLS]∈R r , and input these into the text encoder composed of two layers of transformer. After being processed by the text perceptron, the global text feature T of the [CLS] tag is output CLS (d)∈Rr and local text features T(d)∈R N×r .

[0073] Secondly, the present invention proposes two semantic decomposition modules (Semantic Decomposition Module, referred to as SDM), namely, visual semantic decomposition module and textual semantic decomposition module, to aggregate information from each modality perception feature and decompose them to obtain multi-view semantic embedding. Figure 1 As shown in Figure 1, each semantic decomposition module contains multiple aggregation blocks, each aggregation block contains a cross attention operation, and the perceptual features are optimized through L iterations.

[0074] Specifically, taking the visual semantic decomposition module as an example (the text aspect follows a similar architecture and process), in the initial iteration, the present invention introduces a set of learnable tokens E 0 ∈R K×r , hereinafter referred to as set embedding, where K represents the number of embeddings (K is much smaller than the number of image blocks N). Then, in the t-th aggregation block, t = 1, 2, ..., L, the present invention simultaneously inputs E t-1 and local image features, (x), generate a collection embedding E through cross-attention t The specific operation is as follows: for the tth aggregation block, the present invention maps the local image feature, (x) to the image key variable and image value variables And embed the set into E t-1 Map to Where K is the number of set embeddings. Then, the present invention calculates the dot product between q and k, and then performs a softmax operation to generate an attention map A t ∈R K×r :

[0075]

[0076] A t Then multiply it by the value v, followed by a learnable linear layer W o and residual connection, the result of this process is output as a visual embedding after extracting semantic information

[0077]

[0078] Subsequently, the visual embedding after extracting semantic information is input into a learnable fully connected layer MLP, which promotes the iterative information refinement of the set embedding and obtains the set embedding E output by the t-th aggregation block. t , the formula is:

[0079]

[0080] In the last iteration, the present invention converts the output of the last L-th layer aggregation block (denoted as E L ) and global image features I CLS (x) is combined K times, followed by layer normalization (LayerNorm). This operation produces a multi-view visual semantic embedding with small variance

[0081] B v =LayerNorm(E L +[I CLS ((x))] ×K )

[0082] Similarly, the local text feature T(d) is input into the text semantic decomposition module. After L layers of iterations, the present invention combines the output of the last L-th layer aggregation block with the K repetitions of the global text feature to obtain the multi-view text semantic embedding

[0083] Since the category-level corpus contains many words, it is difficult for the learnable token to focus on different semantic information only through the model architecture. Therefore, the present invention introduces local-to-semantic mapping variance loss and multi-semantic diversity loss to enhance the information decoupling between semantic embeddings and alleviate the feature collapse problem (i.e., multiple semantic embedding representations have the same value). The local-to-semantic mapping variance loss aims to enhance the diversity of set embeddings when aggregating local feature information. This is achieved by forcing the image block (or text tag) to have different attention values ​​on each learnable tag, thereby forcing the set embedding to focus on diverse semantic information. For the similarity matrix between the i-th image block and the set embedding in the t-th aggregation block The present invention formulates the following variance constraints:

[0084]

[0085] Among them, A is the attention matrix of all image patches and set embeddings, L represents the number of layers of the visual semantic decomposition module, M represents the total number of image patches, γ is a constant value used for constraint, and ∈ is used to maintain numerical stability by setting a small scalar, max(0,·) represents the maximum value compared with 0, and Var(·) represents the variance of the set. Finally, the local to semantic mapping variance loss Penalize the variance of each image patch and word token:

[0086]

[0087] in, and are the attention matrices in the text and visual semantic decomposition modules, respectively, and C represents the covariance. After L iterations, each set embedding carries different semantics, laying the foundation for information decoupling.

[0088] At the same time, the present invention introduces a multi-semantic diversity loss to enhance the information decoupling between set embeddings. It enforces the minimum redundancy between set embeddings by making each embedding orthogonal to each other. Since the final semantic embedding contains global features, the orthogonality constraint is invalid. The present invention penalizes the redundancy between the output embeddings of the final aggregation block, i.e., E L To achieve this goal, the present invention normalizes each E L , and then calculate the cosine similarity between them to generate the embedding similarity matrix M = E L (E L ) T ∈R K×K The multi-semantic diversity loss minimizes the similarity matrix close to the identity matrix I∈R by l2-norm. K×K This ensures that different semantic embeddings maintain orthogonality, i.e., the cosine similarity values ​​of the off-diagonal lines approach zero. Multi-semantic diversity loss The formula is:

[0089]

[0090] Among them, M T and M V are the embedding similarity matrices of the text and visual semantic decomposition modules respectively, and K represents the number of elements in the set embedding.

[0091] 3. Partial alignment of document and image space

[0092] After extracting multi-view semantic embeddings, the present invention introduces two loss functions to partially align the semantic information in documents and images at the semantic level and the word-to-image block level respectively.

[0093] like Figure 4 As shown on the right side of , the decomposed semantic alignment loss aims to model the partial associations between multi-view semantic embeddings according to their semantic relevance. The present invention incorporates the Smooth Chamfer similarity into the training process. It assigns different weights to document-image embedding pairs based on similarity, thereby providing dense supervision. The Smooth Chamfer similarity function is:

[0094]

[0095] Among them, LSE(b,B) is a smooth approximation of the maximum cosine similarity between vector b and the elements in set B. sc(x,d) aims to approximate each visual semantic embedding b V and text semantic embedding set B T The smooth alignment between the most similar elements in , while also constraining each text semantic embedding b T and the visual semantic embedding set B V Smooth alignment between the most similar elements in .

[0096] Decomposing Semantic Alignment Loss A cross entropy loss is introduced to encourage images x and corresponding documents d to be closer than other pairs:

[0097]

[0098] Where τ is a temperature parameter, Category documents representing all known categories.

[0099] In addition, the present invention introduces a partial score function S p (x, d), which aims to filter out mismatched information during the inference process to accurately measure semantic similarity. Since each training image only shows part of the semantics described in the document, the present invention limits the similarity calculation to only the document-image semantic pairs with the highest P similarity values, where P is less than the number of semantic embedding sets K. This selective strategy improves the accuracy of semantic similarity by effectively filtering mismatched information between documents and images. The present invention defines the SelectTopCos(x, d, P) function:

[0100] S p (x,d)=S sc (SelectTopCos(x,d,P))

[0101] This function selects the embedding pairs with the highest P similarity values ​​between each visual semantic embedding and all textual semantic embeddings, resulting in a total of P×K pairs of visual-textual embedding pairs. Subsequently, a similar process is applied to each document, selecting each embedding pair with the highest P similarity values ​​between all visual embeddings, resulting in P×P pairs. The smooth chamfer similarity function is then used to calculate the similarity between these selected pairs.

[0102] like Figure 4 As shown on the left side of , the local semantic alignment loss promotes knowledge transfer through fine-grained alignment between image patches and word tokens. The modules involved in this process are as follows Figure 3 As shown, the present invention first obtains local image and local text representations to calculate cross attention. Specifically, three linear layers are used to pass local image features I(x) to output local image query variables Pass the local text feature T(d) to output the local text key variable and local text value variables Then, through the calculation of the attention mechanism, the visual features containing text information are obtained. in is a linear layer that transforms the feature dimension from r h Mapped to the original dimension r. Next, the present invention uses global pooling on the image block dimension v (·) Aggregate fine-grained information to obtain After that, the present invention passes through the linear layer D(·)∈R r×1 Mapping Its output is recorded as the fine-grained similarity score Target local semantic alignment loss is to encourage image x to be in S local (x, d) is close to the category document d of the corresponding label and optimized by cross entropy loss:

[0103]

[0104] in, The class documents representing all known classes. The present invention only uses fine-grained similarity scores in training to save computational cost in the inference process.

[0105] 4. Model training and inference recognition of new category images

[0106] The present invention is optimized by the following loss function:

[0107]

[0108] Among them, λ local , var and λ div is a hyperparameter. Training the above modules together as zero-shot image classification enables the model to generate multi-view disentangled semantic embeddings and accurately model the partial associations between images and documents, thus significantly improving knowledge transfer.

[0109] In the inference phase, given an input image x, the semantic similarity between the current image and all possible categories is calculated, and the predicted category given by the model is is the one that produces the highest partial score S p The unknown category of (x,d′).

[0110]

[0111] in, Represents all unknown categories.

[0112] Experimental test:

[0113] The present invention is tested on three text description-based datasets: a coarse-grained animal classification AWA2 dataset, two fine-grained classification bird classification CUB dataset and flower classification FLO dataset. The present invention uses documents collected from the Internet instead of manually annotated attributes as auxiliary information. For the Zero-Shot Learning setting, the average accuracy of each class in unknown categories is used as the evaluation indicator, denoted as T1. For the Generalized Zero-Shot Learning setting, the average accuracy of each class in known categories (denoted as S) and the average accuracy of each class in unknown categories (denoted as U) and the harmonic mean of the two (denoted as H=2×U×S / (U+S)) are used as evaluation indicators.

[0114] As shown in Table 1, the comparison results of the present invention and the existing text description-based zero-shot image classification methods are shown. For a fair comparison, the present invention and the existing methods use the same text and image perceptrons. The present invention (hereinafter referred to as EmDepart) consistently surpasses the existing methods (I2DFormer and I2MVFormer) in both zero-shot learning settings and generalized zero-shot settings. This proves that the accurate establishment of partial semantic associations in the present invention is important for knowledge. Existing methods do not consider local associations and align the entire semantics of vision and text, so they achieve suboptimal results. In addition, in the setting where only Wikipedia is used as auxiliary information, EmDepart even surpasses the performance of previous methods on AWA2 and FLO datasets when descriptions generated by Wikipedia and large language models are used as auxiliary information at the same time, which further proves that the modeling of partial associations is crucial for text-based zero-shot learning. After adding text descriptions generated based on large language models, the performance of EmDepart is further improved, because the large language model supplements the semantic knowledge for categories with fewer descriptions.

[0115] Table 1: Experimental comparison results of the present invention and the existing method

[0116]

[0117] References:

[0118] [1]Chen,Shiming,et al."Msdn:Mutually semantic distillation networkfor zero-shot learning."Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2022.

[0119] [2]Kong,Xia,et al."En-compactness:Self-distillation embedding&contrastive generation for generalized zero-shot learning."Proceedings of theIEEE / CVF Conference on Computer Vision and Pattern Recognition.2022.

[0120] [3]Liu,Man,et al."Progressive Semantic-Visual Mutual Adaption forGeneralized Zero-Shot Learning."Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2023.

[0121] [4]Su,Hongzu,et al."Distinguishing unseen from seen for generalizedzero-shot learning."Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition.2022.

[0122] [5]Bucher,Maxime,Stéphane Herbin,and Frédéric Jurie."Generatingvisual representations for zero-shot classification."Proceedings of the IEEEInternational Conference on Computer Vision Workshops.2017.

[0123] [6]Kampffmeyer,Michael,et al."Rethinking knowledge graph propagationfor zero-shot learning."Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2019.

[0124] [7]Wang,Xiaolong,Yufei Ye,and Abhinav Gupta."Zero-shot recognitionvia semantic embeddings and knowledge graphs."Proceedings of the IEEEconference on computer vision and pattern recognition.2018.

[0125] [8]Kil,Jihyung,and Wei-Lun Chao."Revisiting document representationsfor large-scale zero-shot learning."arXiv preprint arXiv:2104.10355(2021).

[0126] [9]Al-Halah,Ziad,and Rainer Stiefelhagen."Automatic discovery,association estimation and learning of semantic attributes for a thousandcategories."Proceedings of the IEEE Conference on Computer Vision and PatternRecognition.2017.

[0127]

[10] Zhu,Yizhe,et al."A generative adversarial approach for zero-shotlearning from noisy texts."Proceedings of the IEEE conference on computervision and pattern recognition.2018.

[0128]

[11] Naeem,Muhammad Ferjad,et al."I2dformer:Learning image to documentattention for zero-shot image classification."Advances in Neural InformationProcessing Systems 35(2022):12283-12294.

[0129]

[12] Naeem,Muhammad Ferjad,et al."I2MVFormer:Large Language ModelGenerated Multi-View Document Supervision for Zero-Shot ImageClassification."Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition.2023.

[0130] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.

Claims

1. A zero-shot image classification method, characterized in that: The following steps are involved: 1) Collect web documents and use large language models to generate category description documents: 2) The input image is passed through a visual encoder to generate global and local image features, which are then input into a visual semantic decomposition module for decoupling to generate a visual semantic embedding; at the same time, the category description document corresponding to the image is passed through a text perceptron to generate global and local text features, which are then input into a text semantic decomposition module for decoupling to generate a text semantic embedding; the information decoupling between the visual semantic embedding and the text semantic embedding is enhanced through the local-to-semantic mapping variance loss and the multi-semantic diversity loss; 3) Establish the correlation between visual semantic embedding and text semantic embedding by decomposing the semantic alignment loss, and use the smooth chamfer similarity to measure the semantic similarity between images and documents; Align the semantic information between image patches and document word tokens via local semantic alignment loss; 4) constructing a zero-shot image classification model by using an image sensor, a text sensor, a visual semantic decomposition module, and a text semantic decomposition module, and training the model to optimize the total loss consisting of local-to-semantic mapping variance loss, multi-semantic diversity loss, decomposition semantic alignment loss, and local semantic alignment loss; 5) In the inference stage, the trained zero-shot image classification model is used to calculate the semantic similarity between the input image and all possible categories, and the category with the highest score is selected as the category prediction result.

2. The method according to claim 1, characterized in that The steps in step 1) of using a large language model to generate a category description document include: From collecting web documents related to the category; For categories whose descriptions in documents lack detailed visual information, a large language model is used to generate supplementary visual descriptions through hint templates; The generated supplementary visual descriptions are combined with the collected web documents to obtain category description documents.

3. The method according to claim 1, characterized in that In step 2), the image sensor includes a visual encoder and a residual fully connected layer; the step of generating global and local image features from the image through the visual encoder includes: the visual encoder performs feature encoding on the input image, and the feature encoding is processed through the residual fully connected layer to generate global image features and local image features.

4. The method according to claim 1, characterized in that In step 2), the text perceptron includes a fully connected layer and a text encoder, and the text encoder includes two layers of transformers; the step of generating global and local text features of the document through the text perceptron includes: the text perceptron first initializes the words in the document with GloVe vectors, then reduces the dimension through the fully connected layer processing, and then inputs it into the text encoder to generate global text features and local text features.

5. The method according to claim 1, characterized in that In step 2), the visual semantic decomposition module is decoupled, and the steps of generating visual semantic embedding include: In the initial iteration, a set of learnable tokens is input as a set embedding; In each iteration, the collective embedding and local image features from the previous iteration are input into the aggregation block; Map local image features to image keys k and values ​​v, and map the set embedding to the query q; Calculate the attention map through the cross-attention mechanism; Multiply the attention map with the image value v, and then generate a new visual semantic embedding through a learnable linear layer and residual connection; After multiple iterations, the final visual semantic embedding is obtained.

6. The method according to claim 1, characterized in that In step 2), the text semantic decomposition module is decoupled, and the corresponding steps of generating text semantic embedding include: In the initial iteration, a set of learnable tokens is input as a set embedding; In each iteration, the collective embedding and local text features of the previous iteration are input into the aggregation block; Map local text features to text keys k and values ​​v, and map set embeddings to queries q: Calculate the attention map through the cross-attention mechanism; Multiply the attention map with the text value v, and then generate a new text semantic embedding through a learnable linear layer and residual connection; After multiple iterations, the final text semantic embedding is generated.

7. The method according to claim 1, characterized in that The step of calculating the smooth chamfer similarity in step 3) includes: Compute the similarity between each visual semantic embedding and all textual semantic embeddings; For each visual semantic embedding, select the P pairs of embeddings that are most similar to the textual semantic embedding; for each textual semantic embedding, select the P pairs of embeddings that are most similar to the visual semantic embedding; form a set of P×P pairs of visual-textual embedding pairs; The smooth chamfer similarity function is used to calculate the similarity of the above visual-text embedding pairs to obtain the smooth chamfer similarity between the image and the document.

8. The method according to claim 1, characterized in that The step of calculating the local semantic alignment loss in step 3) includes: passing the local image features to a linear layer to generate a local image query q; Pass the local text features to two linear layers to generate local text keys k and local text values ​​v respectively; Use the local image query q and the local text key k to calculate the similarity between the two and generate an attention mapping matrix; Perform a global pooling operation on the attention mapping matrix and aggregate it to obtain the representation H; A linear layer is used to map the pooled representation H into a fine-grained similarity score, which is used to measure the similarity between the image and the document at the local level. A cross entropy loss function is used to optimize the fine-grained similarity score, resulting in a local semantic alignment loss.

9. The method according to claim 1, characterized in that The total loss function is as follows: Among them, λ local , var and λ div is a hyperparameter, To decompose the semantic alignment loss, is the local semantic alignment loss, is the local to semantic mapping variance loss, is the multi-semantic diversity loss; where τ is a temperature parameter, represents the category documents of all known categories, S SC () represents the smooth chamfer similarity, x represents the image, d represents the corresponding document, d ′ Represents other documents, with no limit on the log base; Where S local () represents the fine-grained similarity score, with an unlimited log base; in Represents the attention matrix in the text semantic decomposition module, A v represents the attention matrix in the visual semantic decomposition module, and C represents the covariance; Among them, M T Represents the embedding similarity matrix of the text semantic decomposition module, M V represents the embedding similarity matrix of the visual semantic decomposition module, I represents the identity matrix, and K represents the number of elements in the set embedding.

10. A zero-sample image classification device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 9 when executing the computer program.

Images