A compositional zero-shot image classification method based on parallel semantic embedding

Abstract

The application discloses a kind of based on parallel semantic embedding's component formula zero sample image classification method, this method first according to the multivariate dependent structure between attribute, object and its combination constructs combination graph, and carries out semantic embedding to combination graph inner node.Subsequently, the combination graph obtained is input into parallel semantic embedding network, and the feature migration between visible class and invisible class is realized by aggregating neighborhood node.Finally, embedding representation obtained by parallel semantic embedding network is combined, and the classification prediction of training image is carried out by joint embedding representation, which is solved with one-hot encoding classification loss, and model parameters are updated and optimized according to classification loss, to obtain optimal model for testing.Test stage will test image input model, and the image category is predicted according to compatibility score.The application adopts parallel semantic embedding mode, improves model generalization, and obtains better component formula zero sample image classification performance.

Description

Technical Field

[0001] This invention relates to the field of image classification, and more specifically to a compositional zero-shot image classification method based on parallel semantic embedding. Background Technology

[0002] Deep learning has become a research hotspot in machine learning in recent years, achieving significant progress in tasks such as image recognition, video action recognition, and text translation. Many supervised learning models based on deep learning have been proposed, such as DenseNet and ArcFace. However, one of the key reasons why deep learning is effective in supervised learning is its need for massive amounts of labeled training data, which often requires a huge amount of human time and effort. Furthermore, the labels on the test dataset must be completely consistent with those on the training dataset; that is, the mappings produced by supervised learning can only handle data of objects of the same category and cannot be transferred to the determination of other categories. While unsupervised learning does not require label supervision, avoiding the complexity and professional limitations of labeling, it cannot provide the category of instances, which contradicts the expectations in practice. More importantly, because data in nature is often long-tailed, meaning that most categories do not have enough suitable training instances, the categories of training instances often fail to cover the test categories. Although some algorithms exist to reduce training samples and improve training efficiency, these algorithms still require a certain number of training samples to train the model on specific classes in order to classify and predict test examples in the test samples.

[0003] Research has found that humans can identify approximately 30,000 object categories and further distinguish between subcategories, such as different car models or different dog breeds. Humans also possess knowledge transfer capabilities, meaning they can transfer previously learned knowledge to new things. Therefore, the concept of zero-shot learning was proposed. Zero-shot learning, meaning learning without training samples, aims to enable deep learning models to recognize new categories that have not been trained on. Initially, the main research method for zero-shot learning used shallow visual features, such as scale-invariant feature transforms (SIFT) and image grayscale histograms, to treat object attributes as semantic space features and then used traditional machine learning methods for discrimination.

[0004] In recent years, a new learning paradigm has emerged called compositional zero-shot learning. In compositional zero-shot learning, both training and test instances are composed of "attribute-object" pairs. The combination to which the test instance belongs has not appeared in the training instances, and the goal of learning is to classify new combinations resulting from the recombination of these "attribute-object" visual units. Therefore, compositional zero-shot learning can be considered a special case of zero-shot learning. Since the annotation spaces corresponding to training and test samples are different in compositional zero-shot learning, it can also be considered a special case of transfer learning.

[0005] The introduction of compositional zero-shot learning offers a solution to the widespread problem in current classification tasks where training categories cannot cover test categories, and also makes it possible to identify data categories that have never been seen before. Classifiers trained under compositional zero-shot learning can not only identify data categories already present in the training set, but also reason and make judgments about data from unseen categories. This enables computers to transfer knowledge, avoids the limitation that training data categories need to cover all test categories, and better meets the needs of real-world production. Summary of the Invention

[0006] The purpose of this invention is to provide a compositional zero-shot image classification method based on parallel semantic embedding. This method utilizes the multi-dependency structure between attributes, objects, and their combinations, and combines multiple networks to obtain a compositional zero-shot image classification model with better performance.

[0007] To achieve the above functions, this invention designs a compositional zero-shot image classification method based on parallel semantic embedding. Through steps S1 to S4, a compositional zero-shot image classification model is constructed and trained. Then, according to steps i to j, the trained compositional zero-shot image classification model is applied to classify the image to be classified.

[0008] Step S1: Construct a preset dataset, which includes images. Obtain all category labels based on the preset dataset. Each image in the preset dataset corresponds to its category label, and each category label is converted into a one-hot encoding.

[0009] Step S2: Construct a compositional zero-shot image classification model based on the multivariate dependencies between attributes, objects, and combinations in a preset dataset. The compositional zero-shot image classification model takes an image as input and outputs a compatibility score representing the image classification probability. The compositional zero-shot image classification model includes a compositional graph construction module, a parallel semantic embedding module, and a classifier learning module. The compositional graph construction module constructs a compositional graph based on the multivariate dependencies between attributes, objects, and combinations in the preset dataset. The parallel semantic embedding module uses a parallel semantic embedding network to obtain feature embedding representations for visible and invisible classes in the preset dataset, respectively. The classifier learning module combines the feature embedding representations of visible and invisible classes to obtain a joint embedding representation. Simultaneously, it uses a feature extractor to extract image features, combines the joint embedding representation and image features to obtain a compatibility function, calculates and outputs the compatibility score.

[0010] Step S3: Construct a training set based on a preset dataset. Each training sample in the training set includes an image, the category label to which the image belongs, and the corresponding one-hot encoding.

[0011] Step S4: Use the above training set to train the compositional zero-shot image classification model, learn the compatibility function between image features and joint embedding representation, until the preset convergence condition is reached, and obtain the trained compositional zero-shot image classification model.

[0012] Step i: Input the image to be classified into the trained compositional zero-shot image classification model to obtain the compatibility function;

[0013] Step j: Calculate the compatibility score of the image to be classified relative to each category label based on the compatibility function, and take the category label with the highest compatibility score as the category to which the image to be classified belongs, thus completing the compositional zero-shot image classification task.

[0014] As a preferred embodiment of the present invention: In step S2, the construction process of the combined graph construction module includes:

[0015] Step S201: Construct a combination graph and a symmetric adjacency matrix based on the attributes, objects, and multivariate dependencies between them in the preset dataset.

[0016] Step S202: Embed node features into the combined graph obtained in step S201 to obtain a node feature matrix.

[0017] As a preferred embodiment of the present invention: the composite diagram in step S201 consists of N nodes. Let A, O, and Y be the attribute set, object set, and concept set, respectively. The attribute set, object set, and concept set are each composed of attribute a, object o, and concept y, respectively. The concept set... Used to describe attributes Object and the dependencies between their combinations; through For each , and The edges of the composite graph are connected, and each node is also connected to itself, resulting in a symmetric adjacency matrix. .

[0018] As a preferred technical solution of the present invention: in step S202, words pre-trained on the large text corpus Wikipedia are selected for node feature embedding to obtain a node feature matrix. , where N is the total number of nodes and P represents the feature dimension of the nodes in the composite graph.

[0019] As a preferred technical solution of the present invention: in step S2, the parallel semantic embedding module is composed of a parallel semantic embedding network; the parallel semantic embedding network includes a graph convolutional network and a graph attention network;

[0020] Graph convolutional networks are based on symmetric adjacency matrices. and node feature matrix The hidden representation of each node is calculated as follows:

[0021] ;

[0022] in, ReLU represents the nonlinear activation function. This represents the hidden representation of the l-th layer. , It is a trainable weight matrix, where U is the output dimension of the upper layer network and V is the output dimension of the current layer network; Given the diagonal node degree matrix, rows in M ​​are normalized to preserve the scale of the feature vectors. The output of the graph convolutional network's output layer is... Obtain the feature embedding representations of the visible and invisible classes: ;

[0023] Graph attention networks are based on symmetric adjacency matrices. and node feature matrix First, calculate the attention cross-correlation coefficient. As shown in the following formula:

[0024] ;

[0025] in, Let i be all the neighboring nodes of node i. Let c represent the nonlinear activation function LeakyReLU. This is the transpose of the model weights. Let Z represent the trainable weight matrix, where Z is the output feature dimension and P represents the feature dimension of the combined graph node. Multi-head attention is used for expansion, and the final output features of each node are as follows:

[0026] ;

[0027] Where g represents the multiplicity of the attention mechanism, and G is the total number of multiplicity of the attention mechanism. Let g be the number of attention cross-correlation coefficients generated by the g-th attention mechanism. Let g be the weight matrix of the g-th attention mechanism; node embedding representation. The input to the graph attention network is N, where N is the number of nodes; the output of the graph attention network is... Obtain the feature embedding representations of the visible and invisible classes: .

[0028] As a preferred embodiment of the present invention: In step S2, the classifier learning module obtains image features F through the feature extractor, and jointly embeds the features of the visible and invisible classes to obtain the joint embedding representation. , The joint weights are calculated based on the joint embedding representation G and image features F, as follows:

[0029] ;

[0030] Where · is the dot product, which calculates the compatibility score of the image relative to each category label based on the compatibility function P.

[0031] As a preferred technical solution of the present invention: step S4, the step of training the compositional zero-shot image classification model includes:

[0032] Step S401: Obtain the feature embedding representations of the visible and invisible classes through the compositional zero-shot image classification model, and further obtain the joint embedding representation. At the same time, obtain the image features through the feature extractor.

[0033] Step S402: Obtain the compatibility score by combining the joint embedding representation with image features;

[0034] Step S403: Calculate the cross-entropy loss between the compatibility score and each category label, calculate the overall loss of the compositional zero-shot image classification model based on the cross-entropy loss, and optimize the parameters of the compositional zero-shot image classification model based on the overall loss value.

[0035] Step S404: Iteratively update the parameters of the compositional zero-shot image classification model using images in the training set until the parameters converge, thus obtaining the trained compositional zero-shot image classification model.

[0036] As a preferred technical solution of the present invention: In step S403, the method for calculating the overall loss of the compositional zero-sample image classification model is as follows:

[0037] Based on the predicted classification probability of the image and the one-hot encoding of the corresponding class label, the classification cross-entropy loss L is calculated. classifier The calculation formula is as follows:

[0038] ;

[0039] In the formula, n is the number of images, and k is the classification probability dimension. It is the value of the i-th dimension after one-hot encoding. The value of the i-th dimension represents the classification probability.

[0040] As a preferred technical solution of the present invention: in step j, the dimension with the largest value in the compatibility function is calculated, and the one-hot encoding corresponding to the category label with the corresponding dimension value of 1 is viewed. The category label corresponding to the one-hot encoding is the category to which the image belongs.

[0041] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0042] 1. The method of this invention utilizes the multi-dependency structure between attributes, objects and their combinations, and explores the combination embedding representation through parallel graph networks to achieve the migration from visible classes to invisible classes;

[0043] 2. The method proposed in this invention improves the generalization performance of the model by combining two embedding representations. Attached Figure Description

[0044] Figure 1 This is a flowchart of the steps of a compositional zero-shot image classification method based on parallel semantic embedding provided in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the framework of a compositional zero-shot image classification model provided in an embodiment of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0047] Reference Figure 1 This invention provides a compositional zero-shot image classification method based on parallel semantic embedding. Through steps S1 to S4, a compositional zero-shot image classification model is constructed and trained. Then, according to steps i to j, the trained compositional zero-shot image classification model is applied to classify the image to be classified.

[0048] Step S1: Construct a preset dataset, which includes images. Obtain all category labels based on the preset dataset. Each image in the preset dataset corresponds to its category label, and each category label is converted into a one-hot encoding.

[0049] Step S2: Construct a compositional zero-shot image classification model based on the multivariate dependencies between attributes, objects, and combinations in a preset dataset. The compositional zero-shot image classification model takes an image as input and outputs a compatibility score representing the image classification probability. The compositional zero-shot image classification model includes a compositional graph construction module, a parallel semantic embedding module, and a classifier learning module. The compositional graph construction module constructs a compositional graph based on the multivariate dependencies between attributes, objects, and combinations in the preset dataset. The parallel semantic embedding module uses a parallel semantic embedding network to obtain feature embedding representations for visible and invisible classes in the preset dataset, respectively. The classifier learning module combines the feature embedding representations of visible and invisible classes to obtain a joint embedding representation. Simultaneously, it uses a feature extractor to extract image features, combines the joint embedding representation and image features to obtain a compatibility function, calculates and outputs the compatibility score.

[0050] The construction process of the composite graph construction module includes:

[0051] Step S201: Construct a combination graph and a symmetric adjacency matrix based on the attributes, objects, and multivariate dependencies between them in the preset dataset.

[0052] In step S201, the composite graph consists of N nodes. Let A, O, and Y be the attribute set, object set, and concept set, respectively. The attribute set, object set, and concept set are each composed of attribute a, object o, and concept y, respectively. The concept set... Used to describe attributes Object and the dependencies between their combinations; through For each , and The edges of the composite graph are connected, and each node is also connected to itself, resulting in a symmetric adjacency matrix. .

[0053] Step S202: Embed node features into the combined graph obtained in step S201 to obtain a node feature matrix.

[0054] Step S202 selects words pre-trained on the large text corpus Wikipedia for node feature embedding. This not only saves feature storage space but also captures rich semantic similarities between words. By adding word embeddings, compositionality is achieved in the semantic space. A node feature matrix is ​​obtained through node feature embedding. , where N is the total number of nodes and P represents the feature dimension of the nodes in the composite graph.

[0055] The parallel semantic embedding module consists of a parallel semantic embedding network; the parallel semantic embedding network includes a graph convolutional network and a graph attention network;

[0056] Graph convolutional networks are based on symmetric adjacency matrices. and node feature matrix The hidden representation of each node is calculated as follows:

[0057] ;

[0058] in, ReLU represents the nonlinear activation function. This represents the hidden representation of the l-th layer. , It is a trainable weight matrix, where U is the output dimension of the upper layer network and V is the output dimension of the current layer network; Given the diagonal node degree matrix, rows in M ​​are normalized to preserve the scale of the feature vectors. The output of the graph convolutional network's output layer is... Obtain feature embedding representations of visible and invisible classes: ;

[0059] Graph attention networks are based on symmetric adjacency matrices. and node feature matrix First, calculate the attention cross-correlation coefficient. As shown in the following formula:

[0060] ;

[0061] in, Let i be all the neighboring nodes of node i. Let c represent the nonlinear activation function LeakyReLU. This is the transpose of the model weights. Let Z represent the trainable weight matrix, where Z is the output feature dimension and P represents the feature dimension of the combined graph node. Multi-head attention is used for expansion, and the final output features of each node are as follows:

[0062] ;

[0063] Where g represents the multiplicity of the attention mechanism, and G is the total number of multiplicity of the attention mechanism. Let g be the number of attention cross-correlation coefficients generated by the g-th attention mechanism. Let g be the weight matrix of the g-th attention mechanism; node embedding representation. The input to the graph attention network is N, where N is the number of nodes; the output of the graph attention network is... Obtain the feature embedding representations of the visible and invisible classes: .

[0064] The classifier learning module obtains image features F through the feature extractor, and jointly embeds the features of the visible and invisible classes to obtain the joint embedding representation. , The joint weights are calculated based on the joint embedding representation G and image features F, as follows:

[0065] ;

[0066] Here, · represents the dot product, which calculates the compatibility score of the image relative to each category label based on the compatibility function P. The compatibility function P assigns a pre-defined high score to correctly matching images and category labels, and a pre-defined low score to incorrectly matching images and category labels; since the compatibility function P contains the compatibility score of each category label and the image, it is also the classification probability of the image.

[0067] Step S3: Construct a training set based on a preset dataset. Each training sample in the training set includes an image, the category label to which the image belongs, and the corresponding one-hot encoding.

[0068] Step S4: Use the above training set to train the compositional zero-shot image classification model, learn the compatibility function between image features and joint embedding representation, until the preset convergence condition is reached, and obtain the trained compositional zero-shot image classification model.

[0069] The steps for training a compositional zero-shot image classification model include:

[0070] Step S401: Obtain the feature embedding representations of the visible and invisible classes through the compositional zero-shot image classification model, and further obtain the joint embedding representation. At the same time, obtain the image features through the feature extractor.

[0071] Step S402: Obtain the compatibility score by combining the joint embedding representation with image features;

[0072] Step S403: Calculate the cross-entropy loss between the compatibility score and each category label, calculate the overall loss of the compositional zero-shot image classification model based on the cross-entropy loss, and optimize the parameters of the compositional zero-shot image classification model based on the overall loss value.

[0073] The overall loss of the compositional zero-shot image classification model is calculated as follows:

[0074] Based on the predicted classification probability of the image and the one-hot encoding of the corresponding class label, the classification cross-entropy loss L is calculated. classifierThe calculation formula is as follows:

[0075] ;

[0076] In the formula, n is the number of images, and k is the classification probability dimension. It is the value of the i-th dimension after one-hot encoding. The value of the i-th dimension represents the classification probability.

[0077] Step S404: Iteratively update the parameters of the compositional zero-shot image classification model using images in the training set until the parameters converge, thus obtaining the trained compositional zero-shot image classification model.

[0078] Step i: Input the image to be classified into the trained compositional zero-shot image classification model to obtain the compatibility function;

[0079] Step j: Calculate the compatibility score of the image to be classified relative to each category label based on the compatibility function, and take the category label with the highest compatibility score as the category to which the image to be classified belongs, thus completing the compositional zero-shot image classification task.

[0080] Calculate the dimension with the largest value in the compatibility function, and look up the one-hot encoding corresponding to the category label with a dimension value of 1. The category label corresponding to this one-hot encoding is the category to which the image belongs.

[0081] The following is an embodiment of the present invention:

[0082] In this embodiment, two compositional zero-shot image classification general datasets are used: MIT-States and C-GQA. MIT-States contains 53,753 everyday images, such as young cats and rusty bikes, with a total of 115 attributes and 245 objects. MIT-States has 1,962 available combinations, of which 1,262 attribute-object pairs are visible during training, and the remaining 700 pairs are not. The C-GQA dataset, with over 9,500 labels, contains 457 attributes and 893 objects, making it the most extensive dataset for CZSL. Specific information about the datasets is shown in Table 1.

[0083] Table 1

[0084]

[0085] The compositional zero-shot image classification model takes images as input and outputs an image compatibility function. It consists of a combinatorial graph construction module, a parallel semantic embedding module, and a classifier learning module. In this embodiment, the compositional zero-shot image classification model uses PyTorch as the deep learning framework, employs the Adam optmizer, has a learning rate of 5e−6 for the feature extractor, and 5e−5 for both GCN and GAT. The batch size is set to 128, and experiments are conducted using an NVIDIA RTX 3090 GPU with 24GB of RAM. The iteration cycle is set to 400 epochs.

[0086] like Figure 2 As shown, the process of training a compositional zero-shot image classification model using a training sample set includes the following steps: The compositional zero-shot image classification model includes a compositional graph construction module, a parallel semantic embedding module, and a joint embedding representation module; the compositional graph construction module constructs a compositional graph based on the attributes, objects, and multivariate dependencies between their combinations in the preset dataset, and embeds semantic features into the nodes in the compositional graph; the parallel semantic embedding module transfers features between visible and invisible classes based on the compositional graph to obtain two types of compositional embedding representations; the classifier learning module combines the above two embedding representations and uses a convolutional neural network pre-trained on the ImageNet dataset as a feature extractor to extract visual features from the input image, obtaining the visual features of the image, and obtaining a compatibility function, i.e., the classification probability, through the image features and the joint embedding representation. The classification probability is then combined with the label one-hot encoding to calculate the cross-entropy loss to obtain the classification loss.

[0087] In this embodiment, experiments were conducted under GZSL settings. During the testing phase, both visible and invisible class samples were classified simultaneously, and the accuracy was compared with three recent mainstream classification methods on various datasets: TMN, SymNet, and CGE. The comparison results are detailed in Table 2. In different metrics, the highest accuracy is indicated in bold, where 1) Seen, visible class accuracy; 2) Unseen, invisible class accuracy; 3) HM, harmonic accuracy of visible and invisible classes; 4) AUC, area under the accuracy curve, quantifying the overall performance of visible and invisible class accuracy under different compensation biases; 5) Atr., Obj., attribute and object accuracy under unbiased compensation. Since the optimal invisible class accuracy is calculated under a large compensation bias, the prediction accuracy for visible classes is 0, only predicting invisible classes, also known as zero-sample performance. When the compensation bias is negative, only visible classes are predicted, and the optimal visible class performance can be calculated. The optimal HM is the highest harmonic accuracy of visible and invisible classes under different compensation biases. As can be seen from Table 2, the method of the present invention is optimal on three commonly used datasets for compositional zero-shot learning, proving the effectiveness of the parallel network method of the present invention.

[0088] Table 2

[0089]

[0090] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A compositional zero-shot image classification method based on parallel semantic embedding, characterized in that, Through steps S1 to S4, a compositional zero-shot image classification model is constructed and trained. Then, according to steps i to j, the trained compositional zero-shot image classification model is applied to classify the image to be classified. Step S1: Construct a preset dataset, which includes images. Obtain all category labels based on the preset dataset. Each image in the preset dataset corresponds to its category label, and each category label is converted into a one-hot encoding. Step S2: Construct a compositional zero-shot image classification model based on the multivariate dependencies between attributes, objects, and their combinations in the preset dataset; The compositional zero-shot image classification model takes an image as input and outputs a compatibility score representing the image classification probability. The compositional zero-shot image classification model includes a combinatorial graph construction module, a parallel semantic embedding module, and a classifier learning module. The combinatorial graph construction module constructs a combinatorial graph based on the central attributes, objects, and multivariate dependencies between their combinations in a preset dataset. The parallel semantic embedding module uses a parallel semantic embedding network to obtain feature embedding representations of visible and invisible classes in a preset dataset, respectively; the classifier learning module combines the feature embedding representations of visible and invisible classes to obtain a joint embedding representation, and at the same time uses a feature extractor to extract image features, and combines the joint embedding representation and image features to obtain a compatibility function, calculates and outputs a compatibility score. In step S2, the parallel semantic embedding module consists of a parallel semantic embedding network; the parallel semantic embedding network includes a graph convolutional network and a graph attention network; Graph convolutional networks are based on symmetric adjacency matrices. and node feature matrix The hidden representation of each node is calculated as follows: ； in, ReLU represents the nonlinear activation function. This represents the hidden representation of the l-th layer. , It is a trainable weight matrix, where U is the output dimension of the upper layer network and V is the output dimension of the current layer network; Given the diagonal node degree matrix, rows in M ​​are normalized to preserve the scale of the feature vectors. The output of the graph convolutional network's output layer is... Obtain feature embedding representations of visible and invisible classes: ; Graph attention networks are based on symmetric adjacency matrices. and node feature matrix First, calculate the attention cross-correlation coefficient. As shown in the following formula: ； in, Let i be all the neighboring nodes of node i. Let c represent the nonlinear activation function LeakyReLU. This is the transpose of the model weights. Let Z represent the trainable weight matrix, where Z is the output feature dimension and P represents the feature dimension of the combined graph node. Multi-head attention is used for expansion, and the final output features of each node are as follows: ； Where g represents the multiplicity of the attention mechanism, and G is the total number of multiplicity of the attention mechanism. Let g be the number of attention cross-correlation coefficients generated by the g-th attention mechanism. Let g be the weight matrix of the g-th attention mechanism; node embedding representation. The input to the graph attention network is N, where N is the number of nodes; the output of the graph attention network is... Obtain the feature embedding representations of the visible and invisible classes: ; Step S3: Construct a training set based on a preset dataset. Each training sample in the training set includes an image, the category label to which the image belongs, and the corresponding one-hot encoding. Step S4: Use the above training set to train the compositional zero-shot image classification model, learn the compatibility function between image features and joint embedding representation, until the preset convergence condition is reached, and obtain the trained compositional zero-shot image classification model. Step i: Input the image to be classified into the trained compositional zero-shot image classification model to obtain the compatibility function; Step j: Calculate the compatibility score of the image to be classified relative to each category label based on the compatibility function, and take the category label with the highest compatibility score as the category to which the image to be classified belongs, thus completing the compositional zero-shot image classification task.

2. The compositional zero-shot image classification method based on parallel semantic embedding according to claim 1, characterized in that, In step S2, the construction process of the composite graph construction module includes: Step S201: Construct a combination graph and a symmetric adjacency matrix based on the attributes, objects, and multivariate dependencies between them in the preset dataset. Step S202: Embed node features into the combined graph obtained in step S201 to obtain a node feature matrix.

3. The compositional zero-shot image classification method based on parallel semantic embedding according to claim 2, characterized in that, In step S201, the composite graph consists of N nodes. Let A, O, and Y be the attribute set, object set, and concept set, respectively. The attribute set, object set, and concept set are each composed of attribute a, object o, and concept y, respectively. The concept set... Used to describe attributes Object and the dependencies between their combinations; through For each , and The edges of the composite graph are connected, and each node is also connected to itself, resulting in a symmetric adjacency matrix. .

4. The compositional zero-shot image classification method based on parallel semantic embedding according to claim 2, characterized in that, In step S202, words pre-trained on the large text corpus Wikipedia are selected for node feature embedding to obtain a node feature matrix. , where N is the total number of nodes and P represents the feature dimension of the nodes in the composite graph.

5. A compositional zero-shot image classification method based on parallel semantic embedding according to claim 2, characterized in that, In step S2, the classifier learning module obtains image features F through the feature extractor, and jointly embeds the features of the visible and invisible classes to obtain the joint embedding representation. , The joint weights are calculated based on the joint embedding representation G and image features F, as follows: ； Where · is the dot product, which calculates the compatibility score of the image relative to each category label based on the compatibility function P.

6. The compositional zero-shot image classification method based on parallel semantic embedding according to claim 1, characterized in that, Step S4, the steps for training the compositional zero-shot image classification model, include: Step S401: Obtain the feature embedding representations of the visible and invisible classes through the compositional zero-shot image classification model, and further obtain the joint embedding representation. At the same time, obtain the image features through the feature extractor. Step S402: Obtain the compatibility score by combining the joint embedding representation with image features; Step S403: Calculate the cross-entropy loss between the compatibility score and each category label, calculate the overall loss of the compositional zero-shot image classification model based on the cross-entropy loss, and optimize the parameters of the compositional zero-shot image classification model based on the overall loss value. Step S404: Iteratively update the parameters of the compositional zero-shot image classification model using images in the training set until the parameters converge, thus obtaining the trained compositional zero-shot image classification model.

7. A compositional zero-shot image classification method based on parallel semantic embedding according to claim 6, characterized in that, In step S403, the overall loss of the compositional zero-shot image classification model is calculated as follows: Based on the predicted classification probability of the image and the one-hot encoding of the corresponding class label, the classification cross-entropy loss L is calculated. classifier The calculation formula is as follows: ； In the formula, n is the number of images, and k is the classification probability dimension. It is the value of the i-th dimension after one-hot encoding. The value of the i-th dimension represents the classification probability.

8. A compositional zero-shot image classification method based on parallel semantic embedding according to claim 6, characterized in that, In step j, the dimension with the largest value in the compatibility function is calculated, and the one-hot encoding corresponding to the category label with a dimension value of 1 is viewed. The category label corresponding to this one-hot encoding is the category to which the image belongs.

Images