An image classification method based on zero-shot learning

Abstract

The present invention discloses an image classification method based on zero-shot learning, which comprises the following steps: obtaining an image set containing images of multiple known categories, wherein each category corresponds to multiple images; annotating each image with a corresponding category label, and obtaining a feature vector of each image, obtaining an attribute vector of the corresponding category according to the annotation of each category label, and obtaining an image sample data set; obtaining an original semantic vector of each category in the image sample data set; constructing an association graph according to the attribute vectors and word vectors of each category based on graph theory, and obtaining a new modal semantic vector of each category; training a basic zero-shot learning model using the image sample data set, wherein the basic zero-shot learning model obtains a semantic vector of each category according to the semantic vectors of each modality of each image and the feature vector of the image; and determining the category of an image to be identified according to the similarity between the feature vector of the image to be identified and the semantic vectors of each category.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and machine learning, and in particular relates to an image classification method based on zero-sample learning. Background Art

[0002] Traditional image recognition models require collecting extensive supervision information for each class of visual objects to be recognized, which is extremely difficult in an open-set setting. Zero-shot learning aims to identify object classes not included in the training data. Unlike traditional fully supervised learning, it does not require labeled data for all classes. By training with labeled data from seen classes and transferring knowledge learned from seen classes to unseen classes using class-specific semantic vectors, zero-shot learning models can mimic human-like recognition of unseen classes.

[0003] As a bridge between images and labels, semantic space can encode knowledge from different perspectives into multi-modal semantic vectors. Semantic vectors play an important role in the process of transferring knowledge from seen categories to unseen categories.

[0004] Semantic vectors, which are widely used today, can be divided into two categories, attribute vectors and word vectors, depending on whether they are manually collected and annotated. Attributes are shareable and nameable visual features of objects obtained through crowdsourcing, while attribute vectors are high-level descriptions of target objects by humans. For example, the AWA2 dataset provides fine-grained annotations for each type of object. These annotations contain binary and real-valued attribute vectors, where each dimension of the vector represents whether a certain attribute appears or how prominent it is in that type of object. Word vectors are vectors representing the meaning of category names obtained through pre-trained models. For example, word2vec uses the CBOW model or the skip-gram model to generate distributed representations of category names. Other semantic vectors are obtained through pre-trained text encoders, such as CLIP, which uses a text encoder to connect text and image features.

[0005] Existing semantic vectors consider each category independently and fail to accurately encode the relationships between categories. For example, in attribute vectors, the relationships between categories are implicitly defined by the annotator. Word vectors are generated using models pre-trained on large-scale corpora. The relationships between categories are implicitly defined in the corpus and then captured by the pre-trained model. Semantic vectors should more accurately encode the relationships between categories.

[0006] Furthermore, the semantic vectors of these different modalities encode knowledge from different perspectives. Attributes are selected by experts, and the annotation process of attribute vectors incorporates human knowledge; word vectors are generated through the model's powerful fitting capabilities and incorporate general knowledge. Integrating knowledge from multiple perspectives helps improve the performance of zero-shot learning models. This knowledge from multiple perspectives suggests that semantic vectors from different modalities are complementary. Existing zero-shot learning methods can be primarily categorized as semantic embedding-based and generation-based. The former constructs a mapping relationship between the image feature space and the semantic space, while the latter uses a generative model to generate image features from semantic vectors and then trains a fully supervised model for classification. Existing methods either utilize a single semantic vector or directly combine semantic vectors from multiple modalities. These efforts fail to fully explore the complementarity between semantic vectors from multiple modalities. Even when fusion methods are employed, the fusion is often incomplete, resulting in inaccurate classification. Summary of the Invention

[0007] Based on the above development status, the purpose of the present invention is to design an image classification method based on zero-shot learning. The present invention trains a zero-shot learning model through multi-level fusion of multimodal semantic vectors to solve the problems that the existing semantic vectors ignore the relationship between categories and the existing zero-shot learning model ignores the complementarity between semantic vectors of different modalities.

[0008] The technical contents of the present invention include:

[0009] An image classification method based on zero-shot learning, comprising the following steps:

[0010] Obtain an image set containing images of multiple known categories, where each category corresponds to multiple images; annotate each image with a corresponding category label, and obtain a feature vector of each image; obtain an attribute vector of the corresponding category based on the category label annotations, and obtain an image sample data set;

[0011] Obtaining an original semantic vector for each category in the image sample dataset; wherein the original semantic vector for category i includes the attribute vector for category i and a word vector generated according to the category label name corresponding to category i;

[0012] Based on graph theory, a first-category association graph is constructed based on the attribute vectors of each category to obtain the third-modal semantic vector of each category. Based on graph theory, a second-category association graph is constructed based on the word vectors of each category to obtain the fourth-modal semantic vector of each category. The attribute vector of a category is called the first-modal semantic vector of the category, and the word vector of the category is called the second-modal semantic vector of the category.

[0013] Using the image sample dataset to train a basic zero-shot learning model, the basic zero-shot learning model obtains a semantic vector for each category based on the four modal semantic vectors of each image and the feature vector of the image;

[0014] For an image to be identified, the category of the image to be identified is determined according to the similarity between the feature vector of the image to be identified and the semantic vectors of each category.

[0015] Furthermore, the basic zero-shot learning model includes a low-level fusion layer, a high-level fusion layer, a fusion module, and an output layer. The basic zero-shot learning model is trained using the image sample dataset to obtain a semantic vector for each category as follows:

[0016] 21) The low-level fusion layer concatenates the four modal semantic vectors of each category y to obtain vector Fusion low (y);

[0017] 22) The advanced fusion layer Fusion the vectors of each category y low (y) By mapping g to the image feature space and performing average pooling on the mapped vector, vector Fusion is obtained. high (y); then vector Fusion low (y) and Fusion high (y) feeding into the fusion module;

[0018] 23) The fusion module first performs vector Fusion low (y) and Fusion high (y) are spliced, and the resulting spliced ​​vectors are linearly mapped and then the attention mechanism is operated. The results obtained by the attention mechanism are then spliced ​​and max-pooled. The max-pooling result is then input into a forward network with residual connection to obtain the semantic vector O(y) of each category y. The loss function value of the image features of the image with category label y and the semantic vector O(y) is then calculated to optimize the basic zero-shot learning model.

[0019] Furthermore, the fusion module is a multi-head self-attention unit with residual connection; the mapping g is a fully connected network with ReLU activation function and 2 hidden layers.

[0020] The loss function is an L2 loss function.

[0021] Furthermore, the basic zero-shot learning model includes an input layer for generating four modal semantic vectors.

[0022] Furthermore, the method for obtaining the third modality semantic vector of each category is:

[0023] a) Based on graph theory, a first category association graph is constructed based on the attribute vectors of each category, where the nodes are categories. If the similarity of the attribute vectors between two categories is greater than a set threshold t, an edge is established between the nodes corresponding to the two categories;

[0024] b) Perform a layer of forward propagation on the first category association graph to obtain the semantic vector matrix Take row m in matrix E as the mth category y m The third modal semantic vector of ; where E′=[φ(y1) T ,φ(y2) T ,…,φ(y C ) T ] T is a matrix composed of attribute vectors of each category, C is the total number of categories, Φ(y C ) is the Cth category y C The attribute vector of P mn is the mth category y m With the nth category y n The similarity value between the elements in the mth row and nth column of matrix A The matrix D is the degree matrix, and the mth row and mth column elements of the matrix D are q is the amplitude of information propagation between control category nodes.

[0025] Furthermore, the method for obtaining the fourth modality semantic vector of each category is:

[0026] a) Based on graph theory, a second category association graph is constructed based on the word vectors of each category, where the nodes are categories. If the similarity of the word vectors between two categories is greater than a set threshold t, an edge is established between the nodes corresponding to the two categories;

[0027] b) Perform a layer of forward propagation on the second category association graph to obtain the semantic vector matrix Take row m in matrix E as the mth category y m The fourth modal semantic vector of ; where E′=[φ(y1) T ,φ(y2) T ,…,φ(y C ) T ] T is a matrix of word vectors of each category, C is the total number of categories, Φ(y C ) is the Cth category y C The word vector of P mn is the mth category y m With the nth category y nThe similarity value between the elements in the mth row and nth column of matrix A The matrix D is the degree matrix, and the mth row and mth column elements of the matrix D are

[0028] Furthermore, the semantic vectors of each category are mapped to the image feature space, and similarity calculations are performed between the obtained vectors and the feature vectors of the image to be identified, and the category of the image to be identified is determined based on the similarity results.

[0029] The present invention first proposes a multi-level fusion method of multimodal semantic vectors for zero-shot learning, which includes:

[0030] Obtain an image sample dataset; the image sample dataset includes image files of all categories (known categories and unknown categories), feature vectors of all images extracted by the pre-trained model, category labels corresponding to all images, attribute vectors of all categories, and seen class / unseen class division.

[0031] Obtain semantic vectors for all categories in the image sample dataset. These include the manually annotated attribute vectors for each category in the dataset, as well as the word vectors for each category obtained by inputting the category label names into a pre-trained text encoder. Semantic vectors specifically refer to the semantic vectors of the category. Each category has only one semantic vector, and each category has multiple images. The features of the images are called feature vectors.

[0032] Based on the two original semantic vectors, we construct a category association graph with the help of graph theory, establish an information propagation mechanism, and obtain a new semantic vector.

[0033] The original semantic vectors and the newly generated semantic vectors are fused at multiple levels. A multi-level fusion mechanism is constructed to fuse the vectors at different steps of zero-shot learning, and a fusion module is further constructed to combine the results of the two. The output of the fusion module is compared with image features to obtain the final zero-shot classification result. The multi-level fusion mechanism and fusion module are used to fully explore the complementarity between semantic vectors and effectively fuse semantic vectors from multiple modalities.

[0034] Furthermore, obtaining semantic vectors for all categories in the dataset includes:

[0035] Normalize the attribute vectors given in the dataset to obtain the attribute vectors for each category. Input each category name into the pre-trained CLIP text encoder and use the output vector as the word vector for that category.

[0036] Furthermore, based on the above two original semantic vectors, with the help of graph theory, new semantic vectors are obtained, including:

[0037] Construct two independent category association graphs. Nodes are categories, and edges are determined based on thresholded similarity between attribute vectors or word vectors between categories. Based on the category association graphs, an information propagation mechanism is constructed to propagate information between different categories, generating new category semantic vectors. These category semantic vectors encode the first-order relationships between categories.

[0038] The graph nodes are categories, and the edges are determined by thresholding the similarity between the original semantic vectors of the categories. First, the similarity matrix P between categories is obtained, where each element φ(y m ) and φ(y n ) represent the categories y m and category y n Attribute vector or word vector of . Threshold each element of the similarity matrix to obtain Then we get the node adjacency matrix A, where each element C represents the total number of categories, q and t respectively control the amplitude of information propagation between category nodes and the number of edges between category nodes. According to graph theory, information propagation between nodes results in a new semantic vector matrix in is the degree matrix, E′=[φ(y1) T ,φ(y2) T ,…,φ(y C ) T ] T is a matrix composed of attribute vectors or word vectors of each category. m That is category y m The new semantic vector of .

[0039] Furthermore, the original semantic vector and the newly generated semantic vector are fused at multiple levels, including:

[0040] Construct a basic zero-shot learning model. This model maps the original semantic vector φ(y) to image features ω(x) using a mapping g. Mapping g is a fully connected network with two hidden layers and a ReLU activation function. During the training phase, mapping g is trained using the image features of the seen class, the semantic vector, and the label. The loss function is the L2 distance between the features of φ(y) mapped by mapping g and the corresponding image features. During the testing phase, the trained mapping g is used to map the semantic vectors of the unseen class into the image feature space. In this image feature space, the test image is classified by comparing the test image features with the mapped features of each category, completing zero-shot learning.

[0041] Concatenate multimodal semantic vectors before mapping and then perform mapping: Fusion low(y)=g([φ a (y),φ w (y),φ′ a (y),φ′ w (y)]), where the element Φ a (y) represents the attribute vector of category y, element Φ w (y) represents the word vector of category y, element Φ′ a (y) represents the new semantic vector obtained based on the attribute vector of category y, element Φ′ w (y) represents the new semantic vector obtained based on the word vector of category y.

[0042] Average pooling of features mapped to image feature space: Fusion high (y)=Avgpool([(g(φ a (y)) T ,g(φ w (y)) T ,g(φ′ a (y)) T ,g(φ′ w (y)) T ] T ).

[0043] Fusion of the above two results low (y) and Fusion high (y) is fed into the fusion module for further fusion to enrich the results of the two. The fusion module is designed as a multi-head self-attention mechanism with residual connections, and its output is O(y).

[0044] Compare the test image features and the output result O(y) after the above fusion of the semantic vectors of each category, and assign the category label with the greatest similarity to the image

[0045] A multi-modal semantic vector multi-level fusion device for zero-shot learning, comprising:

[0046] A data acquisition module is used to acquire an image sample dataset; the image sample dataset includes image features and labels of each category, as well as attribute vectors of each category and seen column / unseen category division;

[0047] Preprocessing module, used to obtain word vectors for each category;

[0048] The new category semantic vector acquisition module is used to construct a category association graph based on attribute vectors and word vectors, and to propagate information on the graph to obtain a new category semantic vector. The new semantic vector encodes the first-order similarity relationship between categories.

[0049] The semantic vector multi-level fusion module is used to fuse semantic vectors of different modalities that encode knowledge from multiple perspectives. The semantic vectors are fused before and after mapping in the zero-shot learning process, and the results of the two are further enriched and combined through subsequent fusion modules.

[0050] The zero-shot learning module is used to classify the test image by comparing the fused semantic vector with the test image features and classifying the test image into the (unknown) category with the highest similarity.

[0051] A computer device, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform any of the above methods.

[0052] A storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when running.

[0053] Compared with the prior art, the present invention has the following positive effects:

[0054] Compared with the prior art, the present invention solves the problem that category semantic vectors ignore the relationship between categories, and the problem that zero-shot learning models ignore the complementarity between semantic vectors of different modalities, thereby improving the accuracy of zero-shot learning. This method first uses graph theory to explicitly model the relationship between categories based on category attribute vectors and word vectors, and constructs a novel information propagation mechanism so that the newly generated category semantic vectors encode the similarity relationship between categories, which is helpful for the knowledge transfer between categories in zero-shot learning, especially the knowledge transfer from seen classes to unseen classes. The present invention proposes an innovative network that fuses different semantic vectors at multiple levels, effectively combines it with the zero-shot learning method, fuses multimodal semantic vectors at different stages of zero-shot learning, and constructs a fusion module with the help of attention mechanism for further fusion, so that knowledge from different perspectives can be fully used in zero-shot learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram for obtaining the new semantic vector in the present invention.

[0056] Figure 2 This is a model architecture diagram of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, scheme and advantages of the present invention more clearly understood, the present invention is further described in detail by taking experiments conducted on real data sets as an example. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] The present invention proposes a multi-level fusion method for multimodal semantic vectors for zero-shot learning. To address the problem that the semantic space in zero-shot learning is not complete enough and knowledge from multiple perspectives cannot be fully utilized for classification, the method considers fusing semantic vectors of multiple modalities at different steps of zero-shot learning to fully utilize the knowledge from different perspectives encoded in the semantic vectors. This will help transfer knowledge from seen categories to unseen categories and improve the accuracy of zero-shot learning.

[0059] Furthermore, this paper considers that existing semantic vectors fail to accurately encode relationships between categories, resulting in an inability to accurately transfer knowledge between categories during zero-shot learning. By leveraging graph theory, we explicitly model the relationships between categories. By injecting information from neighboring category nodes into the current node, we obtain a new semantic vector that encodes the first-order similarity relationships between categories.

[0060] The implementation steps of this plan are as follows:

[0061] 1) Obtain image sample dataset, including image features, labels, attribute vectors (semantic vector of the first modality, φ a (y) is the attribute vector of category y) and the seen / unseen class division. The attribute vector encodes the expert knowledge about the category.

[0062] 2) Input the name of each category into the pre-trained CLIP text encoder to obtain the word vector of each category (the semantic vector of the second modality, φ w (y) is the word vector of category y. The word vector encodes general knowledge about the category.

[0063] 3) Arrange the attribute vectors of each category into a matrix E′=[φ(y1) T ,φ(y2) T ,…,φ(y C ) T ] T . Calculate the cosin similarity between attribute vectors of each category and obtain the similarity matrix P: Thresholding the similarity matrix yields Further get the adjacency matrix This is equivalent to constructing a relationship graph of category nodes and defining whether there is an edge between each two category nodes and the weight of the edge. The initial state of the node is the attribute vector corresponding to the category. Perform a layer of forward propagation on the graph Get a new semantic vector matrix, where row m is category y m The new semantic vector φ′ a (y m ) (Semantic vector of the third modality). This semantic vector encodes the first-order similarity relationship between manually defined categories.

[0064] 4) Arrange the word vectors of each category into a matrix and repeat step 3 to obtain a new semantic vector φ′ w (y m ) (Semantic vector of the fourth modality). This semantic vector encodes a more general first-order similarity relationship between categories.

[0065] 5) Construct a zero-shot classification model that fuses multiple modal semantic vectors. The model is divided into an input layer, a low-level fusion layer, a high-level fusion layer, a fusion module, and an output layer. The input layer needs to prepare the semantic vectors φ of the four modalities mentioned above. a (y),φ w (y),φ′ a (y),φ′ w (y). For the low-level fusion layer, the above semantic vectors are concatenated and mapped to the latent space through the network g: Fusion low (y)=g([φ a (y),φ w (y),φ′ a (y),φ′ w (y)]). The network g is composed of a fully connected layer with an lrelu activation function. For the advanced fusion layer, the four semantic vectors are first mapped to the latent space through the network g, and g(φ a (y))、g(φ w (y)), g(φ′ a (y)), g(φ′ w (y)), and then average pool the output results in the latent space: Fusion high (y)=Avgpool([(g(φ a (y)) T ,g(φ w (y)) T ,g(φ′ a (y)) T ,g(φ′ w (y)) T ] T ). The fusion module further integrates the results of the low-level fusion layer and the high-level fusion layer. It consists of a multi-head attention mechanism and a forward network. First, the fusion results of the two levels are spliced: F(y) = [Fusion low (y),Fusion high (y)], for a certain "head" h in the fusion module, first the concatenated matrix is ​​generated by linear mapping and Then perform the attention mechanism operation The results of multiple "heads" are spliced ​​and max-pooled, and the input of the fusion module is connected to the output of the attention network through a residual connection to obtain the output O′(y)=MaxPool([A1(y),A2(y),…,A H (y)]W O +F(y)). The output is then passed through a feed-forward network with residual connections to obtain the output semantic vector O(y) = r(O′(y)) + O′(y) for each category y of the fusion module.

[0066] 6) For the output layer, during the training phase, for a given image and label pair {x i ,y i}, calculate the L2 distance between the image features and the output of the fusion module, and accumulate the loss on the training set The network parameters are optimized by gradient descent. In the test phase, the network parameters are fixed, and the semantic vectors of all categories are passed through the network in turn. The category with the smallest cosin distance to the current test image feature is found as the label of the current test image.

[0067] 7) For an image to be identified, the category of the image to be identified is determined based on the similarity between the feature vector of the image to be identified and the semantic vectors of each category.

[0068] Taking the AWA2 dataset as an example, the specific implementation steps of the multi-level fusion method of multimodal semantic vectors for zero-shot learning are described.

[0069] The AWA2 dataset collects 37,322 images of 50 animal species. Each animal category has 85-dimensional attribute annotations. 35 animal categories are considered as seen categories, and the remaining 15 animal categories are considered as unseen categories. The goal of zero-shot learning is to train an image classification model on the seen categories and apply it to the task of identifying unseen images.

[0070] The present invention proposes a zero-shot learning method that fuses semantic vectors of multiple modalities for better knowledge transfer from seen classes to unseen classes. First, semantic vectors of multiple modalities are obtained, which are the attribute vectors given in the dataset and the word vectors obtained by inputting the category name into a pre-trained text encoder. Furthermore, through a graph structure, the relationship between categories is explicitly modeled based on the attribute vectors and word vectors, and information is propagated on the graph to obtain two new semantic vectors. The semantic vectors of multiple modalities are passed through a low-level fusion network and a high-level fusion network, and after passing through a fusion module, they are mapped to the image feature space for comparison with image features.

[0071] This embodiment sets up specific experiments to verify the effectiveness of the method in improving the accuracy of zero-shot learning under different settings. In the traditional zero-shot learning setting, in order to avoid the problem of sample category imbalance, the top-1 accuracy of the unseen class classification after macro-average is used for performance testing; in the generalized zero-shot learning setting, the classification accuracy of the seen class is calculated separately. s And the classification accuracy of the unseen class acc u , and then calculate the harmonic mean of the two As a performance evaluation indicator, the mapping network g in this method adopts a two-layer fully connected network with an lrelu activation function, and the hyperparameters are set to p = 0.25, t is set to the mean similarity between the original semantic vectors, and H = 8.

[0072] The output of the model in this embodiment comprehensively considers semantic vectors from multiple modalities. This ensures that when classifying unseen categories, it can combine knowledge from multiple perspectives and more accurately transfer experience learned from seen categories to unseen categories, thereby improving classification accuracy. This embodiment verifies the necessity of fusing semantic vectors from different modalities and the effectiveness of the proposed new semantic vector through multiple ablation experiments. It also verifies the effectiveness of the proposed multi-level fusion network through comparison with other methods.

[0073] The above description is only one embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A zero-shot learning-based image classification method, comprising the following steps: Get an image set containing images of multiple known categories, with each category corresponding to multiple images; Label each image with a corresponding category label, obtain the feature vector of each image, and obtain the attribute vector of the corresponding category based on the category label to obtain an image sample data set; Obtaining an original semantic vector for each category in the image sample dataset; wherein the original semantic vector for category i includes the attribute vector for category i and a word vector generated according to the category label name corresponding to category i; Based on graph theory, a first category association graph is constructed according to the attribute vectors of each category to obtain the third modal semantic vector of each category; based on graph theory, a second category association graph is constructed according to the word vectors of each category to obtain the fourth modal semantic vector of each category; the attribute vector of the category is called the first modal semantic vector of the category, and the word vector of the category is called the second modal semantic vector of the category; wherein, The method for obtaining the third modal semantic vector of each category is as follows: a) Based on graph theory, a first category association graph is constructed according to the attribute vectors of each category, where the nodes are categories. If the similarity of the attribute vectors between two categories is greater than a set threshold t, an edge is established between the nodes corresponding to the two categories; b) A layer of forward propagation is performed on the first category association graph to obtain the semantic vector matrix Take row m in matrix E as the mth category y m The third modal semantic vector of ; where E′=[φ(y1) T ,φ(y2) T ,...,φ(y C ) T ] T is a matrix composed of attribute vectors of each category, C is the total number of categories, Φ(y C ) is the Cth category y C The attribute vector of P mn is the mth category y m With the nth category y n The similarity value between the elements in the mth row and nth column of matrix A The matrix D is the degree matrix, and the mth row and mth column elements of the matrix D are q is the amplitude of information propagation between control category nodes; The method for obtaining the fourth modal semantic vector of each category is as follows: a) Based on graph theory, a second category association graph is constructed according to the word vectors of each category, where the nodes are categories. If the similarity of the word vectors between two categories is greater than the set threshold t, an edge is established between the nodes corresponding to the two categories; b) A layer of forward propagation is performed on the second category association graph to obtain the semantic vector matrix Take row m in matrix E as the mth category y m The fourth modal semantic vector of ; where E′=[φ(y1) T ,φ(y2) T ,...,φ(y C ) T ] T is a matrix of word vectors of each category, C is the total number of categories, Φ(y C ) is the Cth category y C The word vector of P mn is the mth category y m With the nth category y n The similarity value between the elements in the mth row and nth column of matrix A The matrix D is the degree matrix, and the mth row and mth column elements of the matrix D are q is the amplitude of information propagation between control category nodes; The image sample dataset is used to train a basic zero-shot learning model, wherein the basic zero-shot learning model obtains a semantic vector for each category based on the four modal semantic vectors of each image and the feature vector of the image; wherein the basic zero-shot learning model includes a low-level fusion layer, a high-level fusion layer, a fusion module, and an output layer; and the method for obtaining the semantic vector for each category by training the basic zero-shot learning model with the image sample dataset is as follows: 21) The low-level fusion layer concatenates the four modal semantic vectors of each category y to obtain vector Fusion low (y); 22) The advanced fusion layer Fusion the vectors of each category y low (y) By mapping g to the image feature space and performing average pooling on the mapped vector, vector Fusion is obtained. high (y); then vector Fusion low (y) and Fusion high (y) feeding into the fusion module; 23) The fusion module first performs vector Fusion low (y) and Fusion high (y) are concatenated, and then the resulting concatenated vectors are linearly mapped and then the attention mechanism is operated. The results of the attention mechanism are then concatenated and max-pooled. The max-pooled results are then input into a forward network with residual connections to obtain the semantic vector O(y) for each category y. The loss function value between the image features of the image with category label y and the semantic vector O(y) is then calculated to optimize the basic zero-shot learning model. For an image to be identified, the category of the image to be identified is determined according to the similarity between the feature vector of the image to be identified and the semantic vectors of each category.

2. The method according to claim 1, characterized in that The fusion module is a multi-head self-attention unit with residual connection; the mapping g is a fully connected network with ReLU activation function and 2 hidden layers.

3. The method according to claim 1, characterized in that The loss function is an L2 loss function.

4. The method according to claim 1, wherein The basic zero-shot learning model includes an input layer for generating four modal semantic vectors.

5. The method according to claim 1, wherein The semantic vectors of each category are mapped to the image feature space, and the obtained vectors are respectively similarly calculated with the feature vectors of the image to be identified, and the category of the image to be identified is determined based on the similarity results.

6. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images