Zero-shot image classification method, system, device and medium for generating image features with missing attributes

Abstract

A zero-shot image classification method for supplementing missing image features in an image classifier comprises collecting S101 a zero-shot image classification dataset, obtaining semantic features

Description

CROSS-REFERENCE TO RELATED APPLICATIONS 5 This application claims priority to Chinese Patent Application No. 2022115056697, filed on November 28, 2022. The disclosure of the application is incorporated herein for all purposes by reference in its entirety. TECHNICAL FIELD 10 The present disclosure belongs to the technical field of computer vision, in particular to a zero-shot image classification method, a system, a device and a medium for generating image features with missing attributes. BACKGROUND 15 At present, most of the existing image classification models are based on the condition in which all categories of data are known. When the model trained by such data encounters categories of images that does not exist in the training data, the image cannot be recognized. If these images of new categories need to be recognized, it is necessary to re-collect the image data of the new categories and add them to the original dataset, and 20 then re-train the model to allow the model to recognize the new categories. If a new category is encountered again, the above processing steps will be carried out again. With the help of auxiliary information, the zero-shot image classification method transfers the information learned by the model from images of seen categories in the training stage to the categories of images of unseen categories through auxiliary information. 25 The zero-shot image classification methods can be roughly divided into two categories, one is the discriminant zero-shot image classification method, and the other is the generative zero-shot image classification method. The former allows the model to obtain a mapping function from image features to semantic features, and then mapping the test images to a 30 semantic space in a testing stage, the categories of the images are obtained through similarity comparison, which can be considered as solving the zero-shot problem based on metric learning. The latter allows to learn a mapping function from semantic features to 30 09 24 image features, and generate fake image features of unseen categories with semantic features of unseen categories by the learned mapping function, thus solving the problem of zero-shot of unseen categories, and then utilize complete data to train ordinary image classification methods, which can be considered as solving the problem of zero-shot by 5 generating data of unseen categories. Through the above analysis, the problems and defects of the prior art are as follows: The existing generative zero-shot image classification method cannot generate image 10 features that miss some attributes or features, and the distribution of the generated image features does not conform to the real distribution of images of unseen categories, resulting in low classification accuracy of unseen categories. SUMMARY 15 Aiming at the problems existing in the prior art, the present disclosure provides a zero-shot image classification method, a system, a device and a medium for generating image features with missing attributes. The present disclosure is realized as follows: a zero-shot image classification method for 20 generating image features with missing attributes, which comprises: - providing a dataset for a zero-shot image classification, and obtaining category attribute features of all categories in the dataset; - extracting features from images in the dataset to obtain image feature vectors; training a generative adversarial network with the image feature vectors and the category 25 attribute features; - generating fake image features of unseen categories by the generative adversarial network, and combining the fake image features of unseen categories with image feature vectors of seen categories to obtain a combined image training set; - training an image feature classification network model based on the combined image 30 training set; and 30 09 24 - testing data in a test set based on a trained image feature classification network model, wherein the generative adversarial network has a generator, and the category attribute features of the images are input into the generator, and fake image features of unseen 5 categories are output from the generator, wherein by attribute grouping, all values of a group of attributes of category attribute features of unseen categories are set to 0, and then input into the generator to obtain the fake image features of unseen categories lacking the group of attributes, thereby generating the image features with missing attributes. 10 Further, the extracting features from images in the dataset comprising extracting image attribute features corresponding to the images and the image feature vectors obtained by a pre-trained network; and inputting words of each dimension attribute in the image features into Word2vector to obtain a 1024-dimensional image feature vector; the image feature 15 vectors with different attributes are clustered by K-means algorithm, and similar attributes are clustered into one category for attribute grouping. Further, the generative adversarial network further has a discriminator. 20 The fake image features of seen categories and the actual true image features of seen categories extracted through feature extraction are input into the discriminator for determining authenticity, and a confidence of the authenticity of the input features is output, in which true is 1, and false is 0. 25 Further, the generator consists of four layers of neural networks, namely, a fully connected layer of 300x4096, a LeakyReLU activation layer, a fully connected layer of 4096*1024 and a ReLU activation layer; The discriminator consists of four layers of neural networks, namely, a fully connected layer of 1024 x 4096, a LeakyReLU activation layer, a fully connected layer of 4096 x 1 and a sigmoid activation layer. Further, the training formula of the generator is: 5 min L = E[log(l - D(G(a), j))] G where D is the discriminator, G is the generator, a represents the category features, and E represents averaging the dataset; 10 The training formula of the discriminator is: 30 09 24 maxi = E[logD(x,y)] 4- spog (I — 15 where x represents the true image features and x represents the fake image features of unseen categories. Another object of the present disclosure is to provide a zero-shot image classification system for implementing the zero-shot image classification method for generating image 20 features with missing attributes, the zero-shot image classification system comprises: - a dataset module, which is used for collecting a dataset for a zero-shot image classification dataset and obtain category attribute features of all categories in the dataset; - a feature extraction module, which is used for extracting features of images in the dataset to obtain image attribute features and image feature vectors corresponding to the 25 images; - a clustering module, which is used for clustering features by using K-means method to obtain attribute grouping; - a training module, which is used for training the generative adversarial network by 30 09 24 using image feature vectors and category features; - a generative adversarial network module, which is used for generating fake image features of unseen categories, combining the fake image features of unseen categories with image features of seen categories to obtain a combined image training dataset, and 5 training an image feature classification network model by the combined image feature dataset; - a test module, which is used for testing data in a test set based on a trained image feature classification network model, wherein the generative adversarial network has a generator, and the category attribute 10 features of the images are input into the generator, and fake image features of unseen categories are output from the generator, wherein by attribute grouping, all values of a group of attributes of category attribute features of unseen categories are set to 0, and then input into the generator to obtain the fake image features of unseen categories lacking the group of attributes, thereby 15 generating the image features with missing attributes. Another object of the present disclosure is to provide a computer device including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the zero-shot image 20 classification method for generating image features with missing attributes. Another object of the present disclosure is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the zero-shot image classification method for generating image 25 features with missing attributes. Another object of the present disclosure is to provide an information data processing terminal for implementing the zero-shot image classification system for generating image features with missing attributes. 30 09 24 Combined with the technical scheme and the technical problem solved, the technical scheme claimed by the present disclosure has the advantages and positive effects as follows: 5 Firstly, aiming at the technical problems existing in the prior art and the difficulty of solving the problems, combined with the technical solutions to be claimed by the present disclosure and the results and data in the research and development process, etc., how the technical scheme of the present disclosure solves the technical problems and the creative 10 technical effects brought after solving the problems is analyzed in detail and deeply. The specific description is as follows: The method disclosed by the present disclosure belongs to the generative zero-shot image classification method, which optimizes the defect of the existing generative zero-shot 15 image classification method, that is, the situation that the image features missing certain attributes cannot be generated, so that the generated image features of unseen categories are more in line with the actual distribution, which helps the classification model learn more complete information, and finally improve the classification accuracy. 20 Secondly, taking the technical scheme as a whole or from the perspective of products, the technical effects and advantages of the technical scheme to be claimed by the present disclosure are described as follows: The present disclosure utilizes Word2vector to extract semantic features of attributes, thus 25 realizing automatic grouping of attributes; the category attributes are grouped according to the clustering results by K-means algorithm, and then the values of some groups are set to 0 by randomly zeroing when generating the features of unseen categories, so as to get the image features of unseen categories with certain missing features into the input device, thus helping the generated image features to be more in line with the actual distribution. 30 09 24 Does the technical scheme of the present disclosure solve the technical problem that people have been eager to solve, but have never been successful: 5 The present disclosure solves the problem that the feature distribution of the fake image features of unseen categories generated in the existing generative zero-shot image classification method is different from the actual image distribution. BRIEF DESCRIPTION OF THE DRAWINGS 10 Fig. 1 is a flowchart of a zero-shot image classification method for generating image features with missing attributes according to an embodiment of the present disclosure; Fig. 2 shows image data: (a) a image with complete category features; and (b) a image in which partial visual features are missing; 15 Fig. 3 is a schematic structural diagram of the generative adversarial network provided by an embodiment of the present disclosure; Fig. 4 is a schematic diagram of clustering features using a K-means method provided by 20 an embodiment of the present disclosure; Fig. 5 is a process diagram of obtaining category semantic features of missing partial features provided by an embodiment of the present disclosure. 25 DETAILED DESCRIPTION In order to make the object, technical scheme and advantages of the present disclosure clear, the present disclosure is described in further detail below in connection with the embodiments. It should be understood that the specific embodiments described herein are intended to explain the present disclosure only and are not intended to be a limit of the 30 present disclosure. 30 09 24 In order to enable those skilled in the art to fully understand how the present disclosure is specifically implemented, this section describes an explanatory illustration embodiment. 5 A zero-shot image classification method for generating image features with missing attributes according to an embodiment relates to: category attributes are grouped, according to clustering results, by K-means algorithm; and then the values of some groups are set to 0 by randomly zeroing when generating features of unseen categories, so that they are input to an input unit to obtain image features of unseen categories lacking of 10 certain features, thus improving the consistency between the generated image features and the actual distribution. As shown in Fig. 1, the specific process of the zero-shot image classification method for generating image features with missing attributes includes the following steps. 15 S101: Collecting zero-shot image classification dataset and obtaining semantic features of all categories in the dataset; each image in the dataset corresponding to a manually labeled image attribute or feature. 20 S102: Extracting features from each image by a pre-trained network to obtain an image feature vector. S103: Training a generative adversarial network with the image feature vector and the category attributes or features; the generative adversarial network includes two parts: a 25 generator and a discriminator. The category attributes are input into the generator, and fake image features are output. Then, the fake image features and real image features are input into the discriminator, and a confidence of feature is output, e.g., true is 1, and false is 0. S104: Inputting the category features of the unseen categories to the generator of the generative adversarial network, and outputting fake image features of unseen categories. 30 09 24 S105: Combining the fake image features of unseen categories with the image feature vector of real seen categories to obtain a complete image training dataset. 5 S106: Training an image feature classification network, such as ResNet18 image classification network, by complete image training dataset. S107: Testing the trained classification model or network in the test data. 10 In order to demonstrate technical value of technical scheme of the present disclosure, some applications relating to specific products or technologies are illustrated. The entire process of a zero-shot image classification method for generating image 15 features with missing attributes according to an embodiment is as follows: Step 1: Obtaining a zero-shot image classification dataset, CUB bird classification dataset, in which each image corresponds to manually labeled 300-dimensional category features. This dataset has, for example, 11788 images and 200 categories. There are 7057 images 20 in the training set and 4731 images in the test set. It contains 150 seen categories and 50 unseen categories. Each of the 200 categories also has corresponding 300-dimensional category feature. Step 2: Extracting image features (e.g., 1024-dimensional image features) of the image in 25 Step 1 through a ResNet18 network pre-trained on an ImageNet dataset; Step 3: Each dimension of the category features in Step 1 represents an attribute with practical meaning, and a word or words indicating the attribute is / are input into a Word2vector to obtain a semantic vector of the attribute. 30 09 24 Step 4: Clustering the semantic vector in Step 3 by K-means clustering algorithm, with a group of 10. Therefore, the clustering algorithm divides 300 category attributes into 10 groups. 5 Step 5: Training a generative adversarial network by using ordinary or well-known generative zero-shot image classification method, such as CL-WGAN (Xian, Y, Lorenz, T, Schiele, B., &Akata, Z. (2018). Feature generating networks for zero-shot learning, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5542-5551). 10 Step 6: Generating features using a generator of the generative adversarial network in Step 5. In particular, in this step, instead of directly generating features by using the category attributes of unseen categories in Step 1 in the ordinary generative zero-sample image generation method, the present disclosure uses the attribute grouping in Step 4 to 15 set one attribute of the 10 groups of category attributes of unseen categories to 0, and then inputs them into the generator to obtain fake image features of unseen categories with certain attributes missing. At the same time, the complete semantic features of unseen categories are used to generate visual features of unseen categories, as the image features with missing features account for a small part of the overall distribution; 20 Step 7: Training a full-category image feature classifier by utilizing the fake image features of unseen categories generated in Step 6 and the image features of seen categories extracted in Step 2; 25 Step 8: Testing the test set in Step 1 and evaluating. Fig. 2 shows an image with completed category features and an image missing partial visual features; part (a) of Fig. 2 has all the visual features of this category of objects, while part (b) of Fig. 2 has only some features of this category, that is, some visual features are missing. 30 09 24 The general structure of the generative adversarial network trained in this method is shown in Fig.3, comprising: 5 Obtaining the image features of the image by pre-training the model, and input the features to the generator to obtain fake image features, and authenticity is judged by the discriminator. Data are used to train the discriminator and the generator. 10 The training formula for the generator is: min L = £[log(l - D(G(a), y))] G where D is the discriminator, G is the generator, a represents the category features, and E 15 represents averaging the dataset. The training formula of the discriminator is: maxi = EfkjgDfx,y)J 4 Miog fl — D(x,y))] 20 Where x represents the image features and x represents the fake image features. The generator consists of four layers of neural networks, namely, a fully connected layer of 300*4096, a LeakyReLU activation layer, a fully connected layer of 4096*1024 and a ReLLI activation layer; 25 The discriminator also consists of four layers of neural networks, namely, a fully connected layer of 1024 * 4096, a LeakyReLU activation layer, a fully connected layer of 4096 * 1 and a sigmoid activation layer. 30 09 24 Fig. 4 is a schematic diagram of clustering features by K-means method. Category attributes are embodied as words of "round head", "pointed head", "red" and "black" in the figure. After inputting into Word2vector, corresponding 1024-dimensional feature vectors 5 are obtained, and then the feature vectors of different attributes are clustered by K-means to cluster similar attributes into one category. Fig. 5 is the process of obtaining the category semantic features of the missing features by grouping. According to the clustering results, the attributes of the same group are grouped 10 together, and then the value of one random group is assigned to 0, and then the attributes are re-integrated to obtain the category semantic features missing some features, which are then input into the generator. It should be noted that embodiments of the present disclosure may be implemented by 15 hardware software or a combination of software and hardware. The hardware part can be realized by special logic. The software may be stored in memory and executed by a suitable instruction execution system such as a microprocessor or specially designed hardware. Those skilled in the art can understand that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in 20 processor control code, for example, such code is stored on a carrier medium such as a magnetic disk, a CD or DVD-ROM, a programmable memory such as a read-only memory (firmware) or a data carrier such as an optical or electronic signal carrier. The device and its modules of the present disclosure can be realized by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips and transistors, 25 or programmable hardware devices such as field programmable gate arrays and programmable logic devices, or by software executed by various types of processors, or by a combination of the above hardware circuits and software such as firmware. The embodiment of the present disclosure has achieved some positive results in the process of research and development or use, and has great advantages compared with the 30 09 24 prior art. The following contents are described in combination with data, charts and the like of the test process. On CUB dataset, the seen categories Top 1 accuracy, unseen categories Top 1 accuracy 5 and harmonic values of the original CLS-WGAN method are 57.7%, 43.7% and 49.7%, respectively, and the results obtained by the method according to the embodiment of the present disclosure are 58.0%, 50.2% and 53.8%. The method provided by the embodiment of the present disclosure mainly deals with the problem that the generated fake features of unseen categories does not match the actual feature distribution of unseen categories, so 10 the obtained unseen categories Topi accuracy is obviously improved, compared with the original method, thus improving the harmonic value and seen categories are slightly improved. The above is only the particular implementation of the present disclosure, but the protection 15 scope of the present disclosure is not limited to this. Any modification, equivalent substitution and improvement made by any person familiar with the technical field within the technical scope disclosed by the present disclosure within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure. 30 09 24

Claims

1. A zero-shot image classification method for generating image features with missing attributes, comprising:- providing a dataset for a zero-shot image classification, and obtaining category attribute features of all categories in the dataset;- extracting features from images in the dataset to obtain image feature vectors;- training a generative adversarial network with the image feature vectors and the category attribute features;- generating fake image features of unseen categories by the generative adversarial network, and combining the fake image features of unseen categories with image feature vectors of seen categories to obtain a combined image training set;- training an image feature classification network model based on the combined image training set; and- testing data in a test set based on a trained image feature classification network model, wherein the generative adversarial network has a generator, and the category attribute features of the images are input into the generator, and fake image features of unseen categories are output from the generator,wherein by attribute grouping, all values of a group of attributes of category attribute features of unseen categories are set to 0, and then input into the generator to obtain the fake image features of unseen categories lacking the group of attributes, thereby generating the image features with missing attributes.

2. The zero-shot image classification method according to claim 1, wherein said extracting features from images in the dataset comprising extracting image attribute features corresponding to the images and the image feature vectors obtained by a pre-trained network, andwherein each word indicating a corresponding attribute of image attribute features is input into a Word2vector to obtain a 1024-dimensional image feature vector, and the image feature vectors with different attributes are clustered by K-means algorithm such that similar attributes are clustered into one category for attribute grouping.

3. The zero-shot image classification method according to claim 1, wherein the generative adversarial network further has a discriminator; andwherein the discriminator discriminates the authenticity of the image features; the fake image features of seen categories and the actual true image features of seen categories extracted through feature extraction are input into the discriminator for determining30 09 24authenticity, and a confidence of the authenticity of the input features is output, in which true is 1, and false is 0.

4. The zero-shot image classification method according to claim 3, wherein the generator consists of four layers of neural networks, which include a fully connected layer of 300x4096, a LeakyReLU activation layer, a fully connected layer of 4096x1024 and a ReLU activation layer; andwherein the discriminator consists of four layers of neural networks, which include a fully connected layer of 1024x4096, a LeakyReLU activation layer, a fully connected layer of 4096x1 and a sigmoid activation layer.

5. The zero-shot image classification method according to claim 4, wherein the generator has a training formula of:min L = A[log(l - D(G(a), j))] Gwhere D is the discriminator, G is the generator, a represents the category attribute features, E represents averaging the dataset; andthe discriminator has a training formula of the discriminator of:maxL =F[logD(x,y)] +F[log(l-D(x,y))]where x represents the actual image features, and x represents the fake image features of unseen categories.

6. A computer-readable storage medium storing a computer program, the computer program, when executed by a processor, causes the processor to perform the zero-shot image classification method according to any one of claims 1-5.

7. A zero-shot image classification system for implementing the zero-shot image classification method for generating image features with missing attributes according to claim 1, wherein the zero-shot image classification system comprises:- a dataset module for collecting a dataset for a zero-shot image classification and obtaining category attribute features of all categories in the dataset;- a feature extraction module configured to extract features from images in the dataset to obtain image attribute features and image feature vectors corresponding to the images;- a clustering module configured for clustering attribute features by K-means algorithm to obtain attribute groups;- a training module configured fortraining a generative adversarial network with the image feature vectors and the category attribute features;- a generative adversarial network module configured to generate fake image features ofunseen categories, combine the fake image features of unseen categories with the image feature vectors of seen categories to obtain a combined image training set for training an image feature classification network model;- a test module configured to test data in a test set based on a trained image feature classification network model,wherein the generative adversarial network has a generator, and the category attribute features of the images are input into the generator, and fake image features of unseen categories are output from the generator,wherein by attribute grouping, all values of a group of attributes of category attribute features of unseen categories are set to 0, and then input into the generator to obtain the fake image features of unseen categories lacking the group of attributes, thereby generating the image features with missing attributes.30 09 24

Images