A generalized zero-shot image classification method based on fused visual information

Abstract

The application discloses a generalized zero-shot image classification method based on fused visual information, which firstly extracts image feature information by using a pre-trained feature extractor, and adds relative position coding and vectorization to the feature information. Then, the feature vector is input into a feature fusion embedding module, a multi-level architecture is used to differentially process visual information of different resolutions, and fused visual information is obtained by using feature fusion to predict contained semantic attributes. Finally, attribute regression loss and attribute cross-entropy loss are calculated according to the semantic attribute prediction value, model parameter optimization is updated, and an optimal model is obtained for testing. In the testing stage, the test image is input into the model, the semantic attribute combination in the image is obtained, and the image category is predicted according to the cosine similarity score. The application adopts the fused visual information mode, improves the model's information processing capacity for different resolutions, and obtains better zero-shot image classification performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning, and is used for processing image classification problems, and particularly relates to a generalized zero-shot image classification method based on fusion of visual information. BACKGROUND

[0002] Existing deep neural network models often use supervised learning training methods, and the parameters of the model are optimized through a large amount of labeled data training. Labeling all images brings a very high cost, and in the scene of endangered species identification, medical image processing and the like, the lack of training data makes it difficult for the traditional model to achieve good recognition effect on these categories. In order to alleviate the dependence of the model on labeled data in some special tasks, researchers have proposed a new research field-zero-shot learning. In 2009, the AWA dataset was proposed, and then some scholars proposed an algorithm for predicting categories based on attributes, and in the same year, the concept of zero-shot learning was first proposed, which opened the curtain of zero-shot learning.

[0003] Zero-shot, i.e., no training samples, aims to enable a deep learning model to recognize new classes that have not been trained, to realize the situation that the training set and the test set have no intersection in the class of data, or the class of data in the test set is greater than that in the training set, and to classify the images of the invisible class that the model has not been trained through knowledge transfer. Zero-shot learning relies on labeled visible classes and semantic information associated with visible classes and invisible classes. According to the common semantic attributes of the visual features in the image, the attribute features of the visible classes are transferred to the invisible classes, and the coupling relationship between the visible classes and the invisible classes is established under the semantic information associated with the classes, so as to complete the classification of the invisible classes without the label of the invisible class as a training sample. The test stage can be divided into traditional zero-shot learning and generalized zero-shot learning. Traditional zero-shot learning means only predicting the invisible class, and generalized zero-shot learning means predicting both the visible class and the invisible class. Generalized zero-shot learning is more in line with the needs of real tasks, but it also brings greater difficulty and higher requirements for the model.

[0004] Compared with the traditional classification method, the zero-shot learning method has the following advantages:

[0005] 1. For some specific classes (such as endangered species, newly designed industrial products, etc.) for which data sets have not been established, zero-shot learning can be used to realize recognition and classification;

[0006] 2. The implementation method of zero-shot learning has many commonalities with the learning method of human beings, and can improve the explainability of the model;

[0007] 3、Zero-shot learning and deep learning methods can be organically combined and developed to better meet the needs of object recognition tasks.

[0008] The related research of zero-shot learning has high theoretical value and potential application ability, and is one of the inevitable trends of target classification technology development. Zero-shot learning has become one of the research focuses in image recognition classification tasks. SUMMARY

[0009] The purpose of the present application is: in view of the above prior art, the present application proposes a generalized zero-shot image classification method based on fusion visual information, improves the visual feature processing and multi-resolution information interaction capability, and obtains a zero-shot image classification model with better performance.

[0010] To achieve the above purpose, the present application provides the following technical scheme: a generalized zero-shot image classification method based on fusion visual information, through steps S1 to S4, a generalized zero-shot image classification model is obtained, and according to steps i to j, the generalized zero-shot image classification model is applied to classify the image to be classified:

[0011] Step S1, obtaining all class labels, all semantic attributes, and the corresponding relationship between each class label and each semantic attribute based on a preset data set;

[0012] Step S2, constructing a generalized zero-shot image classification model based on all semantic attributes of the preset data set, the generalized zero-shot image classification model takes an image as input and outputs a semantic attribute combination of the image; the generalized zero-shot image classification model includes an image preprocessing module, a feature fusion embedding module and an attribute extraction module; the image preprocessing module obtains a local coding feature vector of the image according to the image; the feature fusion embedding module is used for feature fusion processing of the local coding feature vector to obtain fusion visual information; the attribute extraction module realizes the mapping of the fusion visual information to the preset semantic attribute;

[0013] Step S3, constructing a training set based on visible classes with training samples in the preset data set; each training sample in the training set includes an image, a semantic attribute combination of the image, and a class label to which the image belongs;

[0014] Step S4, training the generalized zero-shot image classification model using the above training set to learn the mapping relationship between the image and the semantic attribute;

[0015] Step i, inputting the image to be classified into the trained generalized zero-shot image classification model to obtain the semantic attribute combination of the image to be classified;

[0016] Step j: Perform cosine similarity calculation on the semantic attribute combination of the image to be classified and the semantic attribute combination under all category labels in the preset dataset. Take the category label corresponding to the semantic attribute combination with the highest similarity as the category label of the image to be classified, so as to realize the generalized zero-shot image classification task.

[0017] Furthermore, in step S2 mentioned above, the image preprocessing module performs image preprocessing steps, including:

[0018] Step S201: Use a feature extractor to extract visual features from the image to obtain a visual feature map of the image;

[0019] Step S202: Divide the visual feature map obtained in step S201 into patches and vectorize each patch. The specific method is as follows: First, reshape the visual feature map of the image to a resolution of 224×224. Then, divide the reshaped visual feature map into regions with a resolution of 4×4 to generate 56×56 non-overlapping regions. After that, vectorize each patch to obtain a 192-dimensional patch feature vector.

[0020] Step S203: Add relative position encoding to the feature vector of each image patch to obtain the local encoded feature vector of the image.

[0021] Furthermore, in the aforementioned step S201, a convolutional neural network pre-trained on the ImageNet dataset is used as a feature extractor.

[0022] Furthermore, in step S203 mentioned above, the step of adding relative position encoding to the tile vector is specifically done by adding a relative position vector to the tile vector. and The relative positional relationships between the blocks can be obtained using the following formula:

[0023]

[0024]

[0025]

[0026] Where x i x j Let d be the input vector. k For the embedding dimension, e ij For x i x j dot product similarity, α ij For e ij x obtained by the softmax function i x j Similarity weights between W QW K W V is a parameter matrix of Query (Q), Key (K) and Value (V) for updating optimization, z i represents the local coding feature vector obtained by processing.

[0027] Further, in the aforementioned step S2, the feature fusion embedding module is composed of DeiT, tile fusion and feature fusion; the tile fusion is used to realize the conversion between different resolutions; the feature fusion adopts a strategy of combination of series and parallel; the first-level DeiT processes the input local coding feature vector, and the processed feature vector is input into the next level while being reconstructed into the first-level feature map by the feature fusion; the feature vector input into the next level is first merged into a 4 times size tile by the tile fusion, the resolution is increased, and then the feature vector is processed by the DeiT of the level, and the obtained feature vector is continuously input into the next level while being reconstructed into the second-level feature map by the feature fusion; through the three tile fusion layers, the model can realize the visual information processing of four resolution levels which are sequentially increased; the feature fusion reconstructs the feature vectors processed by the DeiT of each level into corresponding feature maps, and then the feature maps output by the DeiT of each level are input into the feature fusion layer in parallel, and then the fusion visual information considering local and global is obtained by weighting.

[0028]

[0029]

[0030] wherein, z l-1 is the output feature of the previous layer, is the input feature of the next layer, and z l is the output feature of the next layer.

[0031] Further, in the aforementioned step S2, the attribute extraction module includes a global average pooling layer GAP and a semantic attribute predictor; the semantic attribute predictor is constructed based on all semantic attributes in the preset data set; the fusion visual information is processed by the global average pooling layer GAP to obtain a fusion visual feature map, and then the fusion visual feature map is input into the semantic attribute predictor to obtain corresponding semantic attributes.

[0032] Further, in the aforementioned step S4, the step of training the generalized zero-shot image classification model includes:

[0033] Step S401, inputting the sample image into the generalized zero-shot image classification model to obtain a semantic attribute combination of the sample image, and marking the semantic attribute combination as a predicted semantic attribute combination of the sample image;

[0034] Step S402, obtaining a semantic attribute combination of a category corresponding to the sample image from a preset database based on a category label of the sample image;

[0035] Step S403, calculating a model overall loss by using a loss function based on the semantic attribute combination of the sample image, the predicted semantic attribute combination, and the semantic attribute combination of the corresponding category, and optimizing parameters of the generalized zero-shot image classification model according to the model overall loss value;

[0036] Step S404, iteratively updating the parameters of the generalized zero-shot image classification model by using the sample images in the training set until the parameters converge, to obtain a trained generalized zero-shot image classification model;

[0037] Further, in the foregoing step S403, the calculation of the model overall loss by using the loss function specifically includes:

[0038] calculating an attribute regression loss L based on each single attribute in the semantic attribute combination of the sample image and the predicted semantic attribute combination AR , and the calculation formula is as follows:

[0039]

[0040] In the formula, M is a total number of semantic attributes included in a preset data set, is a value of each single attribute in the predicted semantic attribute combination, a i is a value of each single attribute in the semantic attribute combination of the sample image;

[0041] calculating an attribute cross-entropy loss L based on the predicted semantic attribute combination of the sample image and the semantic attribute combination of the corresponding category ACE , and the calculation formula is as follows:

[0042]

[0043] In the formula, y s is a visible class training set used, is an attribute combination included in the visible class;

[0044] calculating the model overall loss L based on the attribute regression loss L AR and the attribute cross-entropy loss L ACE , and the calculation formula is as follows:

[0045] L = L AR + aL ACE

[0046] In the formula, a is a weighting coefficient between two losses.

[0047] Further, in the aforementioned step j, the specific formula of the cosine similarity calculation is as follows:

[0048]

[0049] In the formula, represents the attribute combination contained in the contrast category (including the visible category and the invisible category).

[0050] The generalized zero-shot image classification method based on fused visual information has the following technical effects compared with the prior art by adopting the above technical scheme:

[0051] 1. The method adopts effective visual information fusion to improve the model's processing ability for different resolution level information, so that the performance of the zero-shot learning classification model is more optimal and the precision is higher.

[0052] 2. The method fuses visual information of multiple resolutions, so that the model takes into account the global analysis ability and the local feature extraction ability of the image, has good visual-semantic embedding effect, and realizes the migration from the visible category to the invisible category. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a step flow chart of the generalized zero-shot image classification method based on fused visual information proposed by the present application;

[0054] Figure 2 is a frame schematic diagram of the feature fusion embedding module proposed by the present application;

[0055] Figure 3 is an image processing schematic diagram of the generalized zero-shot image classification model based on fused visual information in the embodiment. DETAILED DESCRIPTION

[0056] In order to better understand the technical content of the present application, specific embodiments are described below with reference to the accompanying drawings.

[0057] Aspects of the present application are described in this detailed description and illustrated in the accompanying drawings by various illustrative embodiments. The embodiments of the present application are not limited to the drawings described. It should be understood that the present application is implemented by any one of the above-mentioned concepts and embodiments, and the concepts and embodiments described in detail below, because the disclosed concepts and embodiments of the present application are not limited to any embodiment. In addition, some aspects disclosed by the present application can be used alone or in any suitable combination with other aspects disclosed by the present application.

[0058] As Figure 1As shown, the present application proposes a generalized zero-shot image classification method based on fused visual information, which includes the following steps:

[0059] (1) Obtain all class labels, semantic attributes, and their corresponding relationships of the dataset;

[0060] (2) Construct a generalized zero-shot image classification model;

[0061] (3) Construct a visible class training sample set;

[0062] (4) Train the generalized zero-shot image classification model with the training sample set;

[0063] (5) Input the image to be classified into the trained model to obtain the corresponding semantic attribute combination;

[0064] (6) Compare the obtained semantic attribute combination with the semantic attribute combinations of all classes, and output the class label with the highest similarity.

[0065] In this embodiment, three generalized zero-shot image classification general datasets are used respectively: AWA2, CUB, and SUN. Among them, AWA2 is an animal dataset containing 37322 images, 50 categories, and 85 semantic attributes. CUB is a bird recognition dataset containing 11788 images, 200 categories, and 102 semantic attributes. SUN is a place recognition dataset containing 14340 images, 717 categories, and 312 semantic attributes. Each sample data contains an image, a class label, and a corresponding semantic attribute combination. The specific information of the dataset is shown in Table 1:

[0066] Table 1

[0067] Dataset Training samples Visible / Invisible test samples Visible / Invisible classes Semantic attributes AWA2 23527 5882 / 7913 40 / 10 85 CUB 7057 1440 / 2580 150 / 50 102 SUN 10320 7924 / 1483 645 / 72 312

[0068] The generalized zero-shot image classification model takes an image as input and outputs the semantic attribute combination of the image, which is composed of an image preprocessing module, a feature fusion embedding module, and an attribute extraction module. In this embodiment, the generalized zero-shot image classification model uses pytorch as a deep learning framework, uses Adam optmizer to optimize parameters with a fixed learning rate of 0.0001, sets the batch size to 64, sets the loss weight coefficient to 0.01, uses NVIDIA RTX 3090 GPU 24GB for experiments, and sets the iteration rounds to 100 epochs.

[0069] As Figure 2As shown, the feature fusion embedding module includes DeiT, tile fusion, and feature fusion; tile fusion realizes conversion between different resolutions, and tile fusion is used before DeiT except for the first layer to increase the resolution of each level in turn, so that different levels of DeiT perform self-attention calculation on different resolution tile vectors; the feature fusion part adopts a fusion strategy combining series and parallel, maintaining the series structure between different levels while parallel inputting the feature maps output by each level into the feature fusion layer for fusion. Wherein, DeiT is composed of multi-head self-attention layer MSA and MLP, layer normalization is performed before MSA and MLP, and then residual connection is performed, and the network relationship can be represented by the following formula:

[0070]

[0071]

[0072] In the formula, z l-1 is the output feature of the previous layer, is the input feature of the subsequent layer, and z l is the output feature of the subsequent layer.

[0073] The attribute extraction module includes a global average pooling layer GAP and a semantic attribute predictor; the semantic attribute predictor is constructed based on all semantic attributes in each corresponding data set, the fusion visual feature map is obtained by processing the fusion visual information through the global average pooling layer GAP, and then the fusion visual feature map is input into the semantic attribute predictor to obtain the corresponding semantic attribute combination.

[0074] As Figure 3 shown, the process of training the generalized zero-shot image classification model using the training sample set includes the following steps: using a convolutional neural network pre-trained on the ImageNet data set as a feature extractor to perform visual feature extraction on the input image, obtaining the visual feature map of the image; then the visual feature map is uniformly reshaped to 224x224 resolution, and the reshaped visual feature map is divided into 56x56 non-overlapping region tiles with 4x4 resolution as a region tile, and then each tile is vectorized to obtain a 192-dimensional tile feature vector; finally, relative position encoding is added to each tile feature vector to obtain the local encoding feature vector of the image. Wherein, the specific method of adding relative position encoding is: adding a relative position vector and The relative position relationship between the tiles is obtained by the following formula:

[0075]

[0076]

[0077]

[0078] Where x i x j Let d be the input vector. k For the embedding dimension, e ij For x i x j dot product similarity, α ij For e ij x obtained by the softmax function i x j Similarity weights between W Q W K W V To update and optimize the query (Q), key (K), and value (V) parameter matrix, z i This represents the local encoded feature vector obtained after processing.

[0079] The local encoded feature vectors of the image are input into the feature fusion and embedding module for feature fusion and embedding. The first-level DeiT processes the input local encoded feature vectors, and the processed feature vectors are input into the next level and simultaneously reconstructed into the first-level feature map through feature fusion. The feature vectors input into the next level first undergo patch fusion to merge four adjacent patches into one patch of 4 times the size, increasing the resolution. Then, they are processed by the DeiT at this level, and the resulting feature vectors are input into the next level. At the same time, they are reconstructed into the second-level feature map through feature fusion. Through these three patch fusion layers, the model can process visual information at four progressively increasing resolution levels. Feature fusion reconstructs the feature vectors processed by each level of DeiT into corresponding feature maps, and then inputs the feature maps output by each level of DeiT into the feature fusion layer in parallel. Finally, they are weighted to obtain fused visual information that takes into account both local and global information.

[0080] The semantic attribute prediction is performed by the visual information input attribute extraction module to obtain the predicted semantic attribute values. These predicted values ​​are then compared with the true semantic attribute values ​​of the image to calculate the overall model loss. Based on this overall model loss, the parameters of the generalized zero-shot image classification model are optimized. The method for calculating the overall model loss is as follows:

[0081] Based on the semantic attribute combination of the sample image and each individual attribute in the predicted semantic attribute combination, the attribute regression loss L is calculated. AR The calculation formula is as follows:

[0082]

[0083] In the formula, M represents the total number of semantic attributes contained in the preset dataset. To predict the value of each individual attribute in a combination of semantic attributes, ai for each single attribute in the semantic attribute combination of the sample image;

[0084] Based on the predicted semantic attribute combination of the sample image, the semantic attribute combination of the corresponding category, the attribute cross-entropy loss L is calculated ACE , and the calculation formula is as follows:

[0085]

[0086] In the formula, y s is the visible class training set used, is the attribute combination contained in the visible class;

[0087] Based on the attribute regression loss L AR and the attribute cross-entropy loss L ACE , the overall loss L of the model is calculated, and the calculation formula is as follows:

[0088] L=L AR +αL ACE

[0089] In the formula, a is the weighting coefficient between the two losses.

[0090] The training samples in the AWA2, CUB and SUN data sets are used to train the model, and the parameters of the generalized zero-shot image classification model are iteratively updated until the parameters converge, and the trained generalized zero-shot image classification model is obtained.

[0091] The image to be classified is input into the trained generalized zero-shot image classification model, and based on the mapping relationship between the image generated by the generalized zero-shot image classification model training and each semantic attribute, the semantic attribute combination corresponding to the image is obtained. The obtained semantic attribute combination and the semantic attribute combination of all categories in the corresponding data set are respectively calculated by cosine similarity, and the category label corresponding to the semantic attribute combination with the highest similarity is taken as the classification result output of the generalized zero-shot image classification model. The specific formula of the cosine similarity calculation is as follows:

[0092]

[0093] In the formula, represents the attribute combination contained in the contrast category (including the visible class and the invisible class).

[0094] In the embodiment, experiments are carried out under the GZSL setting, the test stage simultaneously classifies visible class and invisible class samples, and the precision is compared with three recent mainstream classification methods, that is, the generative adversarial zero-shot learning method GAZSL, the semantic preserving adversarial embedding network SP-AEN and the ViT-based zero-shot learning method ViT-ZSL. The comparison results are shown in Table 2. In different indexes on each data set, the highest precision is represented in bold, wherein Acc S , Acc U respectively represent the Top-1 precision of class prediction for visible class and invisible class samples, Acc H represents the harmonic mean of Acc S and Acc U , and represents the comprehensive resolution performance of the model for two categories of prediction; from Table 2, it can be seen that the classification precision of the method of the present application on three commonly used data sets of zero-shot learning reaches the highest, which proves the effectiveness of the visual information fusion adopted by the method of the present application. And the comprehensive index Acc H of the three data sets is improved by 1.2%, 3.5% and 6.3% compared with the second best method, which shows that the method of the present application achieves a good balance in the prediction of visible class and invisible class, and the overall precision of the model is superior to other recent zero-shot learning models.

[0095] Table 2

[0096]

[0097] Although the present application has been described as above with reference to the preferred embodiments, it is not intended to limit the present application. Those skilled in the art having ordinary knowledge in the technical field of the present application can make various modifications and decorations without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application shall be subject to the definition of the claims.

Claims

1. A generalized zero-shot image classification method based on fusion of visual information, for classifying images of unseen classes that the model has not been trained on, characterized in that, By steps S1 to S4, a generalized zero-shot image classification model is obtained, and by steps i to j, the generalized zero-shot image classification model is applied to classify the to-be-classified image: Step S1, obtaining all class labels, all semantic attributes, and the correspondence between each class label and each semantic attribute based on a preset data set; Step S2, constructing a generalized zero-shot image classification model based on all semantic attributes of the preset data set, the generalized zero-shot image classification model taking an image as input and outputting a semantic attribute combination of the image; the generalized zero-shot image classification model includes an image preprocessing module, a feature fusion embedding module, and an attribute extraction module; the image preprocessing module obtains a local coding feature vector of the image according to the image; The feature fusion embedding module is used for feature fusion processing on the local coding feature vector to obtain fused visual information; the attribute extraction module realizes the mapping of the fused visual information to the preset semantic attributes; the feature fusion embedding module includes DeiT, tile fusion, and feature fusion; tile fusion realizes conversion between different resolutions, and uses tile fusion before DeiT except for the first layer to increase the resolution of each level in turn, realizing self-attention calculation of different levels of DeiT on different resolution tile vectors; the feature fusion part adopts a fusion strategy combining series and parallel to maintain the series structure between different levels while parallelly inputting the feature maps output by each level into a feature fusion layer for fusion; Step S3, constructing a training set based on visible classes with training samples in the preset data set; Each training sample in the training set includes an image, a semantic attribute combination of the image, and a class label to which the image belongs; Step S4, training the generalized zero-shot image classification model using the training set to learn the mapping relationship between the image and the semantic attribute; Step i, inputting the to-be-classified image into the trained generalized zero-shot image classification model to obtain a corresponding semantic attribute combination; step j, performing cosine similarity calculation on the semantic attribute combination of the to-be-classified image and the semantic attribute combinations under all class labels in the preset data set, taking the class label corresponding to the semantic attribute combination with the highest similarity as the class label of the to-be-classified image to realize the generalized zero-shot image classification task.

2. The generalized zero-shot image classification method based on fused visual information according to claim 1, wherein, In step S2, the steps of the image preprocessing module for preprocessing the image include: Step S201, extracting visual features of the image using a feature extractor to obtain a visual feature map of the image; Step S202, dividing the visual feature map obtained in step S201 into tiles and vectorizing each tile; the specific method is: first, uniformly reshape the visual feature map of the image to a resolution of 224x224, then divide the reshaped visual feature map into 56x56 non-overlapping region tiles at a resolution of 4x4, and then vectorize each tile to obtain a 192-dimensional tile feature vector; Step S203, adding relative position coding to each tile feature vector to obtain a local coding feature vector of the image.

3. The generalized zero-shot image classification method based on fused visual information according to claim 2, characterized in that, In the step S201, a convolutional neural network pre-trained on an ImageNet dataset is used as a feature extractor.

4. The generalized zero-shot image classification method based on fused visual information according to claim 2, characterized in that, The step S203 adds a relative position code to the tile vector, and the specific method is as follows: adding a relative position vector to the tile vector and The relative position relationship between the tiles is obtained by the following formula: Where x i x j Let d be the input vector. k For the embedding dimension, e ij For x i x j dot product similarity, α ij For e ij x obtained by the softmax function i x j Similarity weights between W Q W K W V To update and optimize the query (Q), key (K), and value (V) parameter matrix, z i This represents the local encoded feature vector obtained after processing.

5. The generalized zero-shot image classification method based on fused visual information according to claim 1, wherein, In the step S2, the DeiT is composed of a multi-head self-attention layer MSA and an MLP, and the MSA and the MLP are subjected to layer normalization processing before residual connection, and the network relationship can be represented by the following formula: where z l-1 is the output feature of the preceding layer, is the input feature of the succeeding layer, z l is the output feature of the succeeding layer; The first-level DeiT processes the input local encoding feature vector, and the processed feature vector is input into the next level while being reconstructed into the first-level feature map through feature fusion. The feature vector input into the next level is first merged into a 4 times size block by block fusion, and the resolution is increased, and then processed by the DeiT of the level, and the processed feature vector is continuously input into the next level while being reconstructed into the second-level feature map through feature fusion. Through the three block fusion layers, the model can process visual information of four resolution levels in turn. Feature fusion reconstructs the feature vectors processed by the DeiT of each level into corresponding feature maps, and then the feature maps output by the DeiT of each level are input into the feature fusion layer in parallel, and then weighted to obtain the fusion visual information considering both local and global.

6. The generalized zero-shot image classification method based on fused visual information according to claim 1, wherein, In the step S2, the attribute extraction module includes a global average pooling layer GAP and a semantic attribute predictor. The semantic attribute predictor is constructed based on all semantic attributes in a preset dataset. The fusion visual information is processed by the global average pooling layer GAP to obtain a fusion visual feature map, and then the fusion visual feature map is input into the semantic attribute predictor to obtain the corresponding semantic attribute.

7. The generalized zero-shot image classification method based on fused visual information according to claim 1, wherein, In the step S4, the step of training the generalized zero-shot image classification model includes: Step S401, inputting a sample image into the generalized zero-shot image classification model to obtain a semantic attribute combination of the sample image, and marking the semantic attribute combination as a predicted semantic attribute combination of the sample image; Step S402, obtaining a semantic attribute combination of the corresponding category of the sample image from a preset database based on the class label of the sample image; Step S403, based on the semantic attribute combination, the predicted semantic attribute combination and the semantic attribute combination of the corresponding category of the sample image, using a loss function to calculate the overall loss of the model, and optimizing the parameters of the generalized zero-shot image classification model according to the overall loss value of the model; Step S404, iteratively updating the parameters of the generalized zero-shot image classification model using the sample images in the training set until the parameters converge, and obtaining the trained generalized zero-shot image classification model.

8. The generalized zero-shot image classification method based on fused visual information according to claim 7, characterized in that, In step S403, the overall loss of the model is calculated using a loss function, which specifically includes: Based on the semantic attribute combination of the sample image and each single attribute in the predicted semantic attribute combination, an attribute regression loss L is calculated AR The calculation formula is as follows: In the formula, M is the total number of semantic attributes contained in the preset data set, a is the value of each single attribute in the semantic attribute combination of the sample image, i a is the value of each single attribute in the semantic attribute combination of the sample image, Based on the predicted semantic attribute combination of the sample image, the semantic attribute combination of the corresponding category, the attribute cross-entropy loss L is calculated ACE ; Based on the attribute regression loss L AR , the attribute cross-entropy loss L ACE , the overall loss L of the model is calculated, and the calculation formula is as follows: L = L AR + aL ACE In the formula, α is the weighting coefficient between the two losses.

9. The generalized zero-shot image classification method based on fused visual information according to claim 8, wherein, In the step j, the specific formula of the cosine similarity calculation is as follows: In the formulae, represents the attribute combination contained in the contrast category.

Images