An image augmentation model training method and image classification method based on variational autoencoder and adversarial generative network

Abstract

The present invention discloses an image augmentation model training method and an image classification method based on a variational autoencoder and a generative adversarial network. The method inputs the visual features and semantic features of a visible class training image into the visual modality and semantic modality variational autoencoders to generate a first pseudo visual feature and a pseudo semantic feature respectively, and inputs them into a generator network to generate a second pseudo visual feature, and then uses a discriminator network to discriminate between real features and generated features, and uses a visible class training image data set to train the variational autoencoder generative adversarial network model. For zero-shot image classification, the model trained on the visible class generates pseudo visual features of unseen class training images and combines the class label training classifier to classify the unseen class images; it can effectively fuse the visual information and semantic information of the image, generate visible class and unseen class images that are closer to the real data distribution and of high quality, and improve the accuracy of zero-shot image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to an image augmentation model training method and an image classification method based on a variational autoencoder and a generative adversarial network. Background Art

[0002] Traditional image classification tasks not only require a large amount of labeled image data, but also have poor performance when the categories of the model training set and the test set are inconsistent. For example, for an unseen picture or a picture that does not belong to any category in the training set, it is necessary to collect samples again and annotate them, and obtain enough training samples to retrain the model so that the model can recognize the picture. This process is not only costly and slow, but also in reality, the collection and labeling of a large number of annotated images are highly complex and uncertain. Therefore, in order to solve the problem of missing samples of unseen classes, zero-shot learning (ZSL) was proposed.

[0003] Zero-shot learning is a special scenario of transfer learning, which is used to solve the problem of identifying samples of unseen classes in training samples. In layman's terms, zero-shot learning is to enable the model to simulate the way people reason and recognize things that have never been seen. Samples with labels in the feature space are visible classes, and samples without labels in the feature space are unseen classes. Traditional zero-shot learning aims to find the mapping relationship between image visual features and semantic features from a given visible class image, and then generalize it to unseen class images, recognize unseen class images, and thus achieve zero-shot image recognition tasks. For example, using cauliflower image data to train a zero-shot recognition model, and inputting the semantic relationship "broccoli is green cauliflower" into the zero-shot model, the model can recognize and classify broccoli images.

[0004] To achieve zero-shot learning, we must first establish two most basic spaces: feature space and category semantic space. The elements in the feature space are the visual features of all images, and the category semantic space is a description of the attributes of the image label, generally expressed as a semantic attribute space or a semantic word vector space; what zero-shot learning needs to do is to learn the mapping relationship between the feature space and the semantic space. Usually, the visual features in the feature space are extracted through a deep convolutional neural network, which has a higher dimension, while the dimension in the semantic space is lower, so the mapping relationship between the feature space and the semantic space is divided into the mapping from the feature space (high-dimensional space) to the semantic space (low-dimensional space) and the mapping from the semantic space (low-dimensional space) to the feature space (high-dimensional space). The mapping from high-dimensional space to low-dimensional space is finally realized. For any image, the mapping relationship is learned by neural network to map the unknown feature from feature space to semantic space, and then a nearest neighbor is found in the semantic space to realize the recognition and classification of the image at one time; the mapping from low-dimensional space to high-dimensional space is finally realized. In the case of unseen class images, its semantic information is described, and its semantic features are obtained through the word vector model, which has a lower dimension. Then, the image features of the unseen class are generated through the learned mapping relationship, and the features are input into the classifier to obtain the category to which the features belong. However, due to the different data distribution between the visible class and the unseen class, directly mapping the visual space and the semantic space will cause the recognition of the unseen class to be biased towards the visible class. In order to alleviate these problems, a generative model for zero-shot learning is proposed. Specifically, the visual features and semantic features of the unseen class samples are input into the generative model to synthesize training samples to directly train a classifier, converting the zero-shot classification into classic supervised learning.

[0005] The generative models for zero-shot learning are mainly based on generative adversarial networks (GAN) and variational autoencoders (VAE). The samples generated by generative adversarial networks are clearer and more realistic, but the training network of generative adversarial networks is unstable, which causes some generated samples to deviate seriously from the true distribution and the model is prone to collapse. Unlike generative adversarial networks, the training process of variational autoencoders is relatively stable, and the difference between reconstructed images and original images is directly compared. However, variational autoencoders directly calculate the mean square error between the reconstructed images and the original images as the loss function, which makes the generated images of lower quality. Summary of the invention

[0006] 1. Problems to be solved

[0007] In order to solve the deficiencies in the above-mentioned prior art, the present invention aims at the problem of zero-shot image classification, combines the advantages of variational autoencoders and generative adversarial networks, fully integrates the visual information and semantic information of the image to generate more effective samples, and proposes an image augmentation model training method based on variational autoencoders and generative adversarial networks for effectively synthesizing the visual features of unseen images. The method combines a variational autoencoder with a stable training process with a generative adversarial network with clear generated samples, and simultaneously inputs the visual features and semantic features of the image sample to effectively match the visual information and semantic information of the image, improve the quality of the generated data, and effectively solve the problem of missing unseen images in zero-shot learning. The generated pseudo-samples are used to train the classifier, and the zero-shot learning is converted into classic supervised learning, thereby improving the accuracy of zero-shot image classification.

[0008] 2. Technical solution

[0009] To solve the above problems, the present invention adopts the following technical solutions:

[0010] A first aspect of the present invention provides an image augmentation model training method based on a variational autoencoder and a generative adversarial network, characterized in that the method comprises the following steps:

[0011] S110: Obtain a visible class training image, and extract visual features and semantic features of the visible class training image;

[0012] S120: pre-configuring an image augmentation model, wherein the image augmentation model includes a visual modality variational autoencoder, a semantic modality variational autoencoder, and a generator configured according to a generative adversarial network;

[0013] S130: Inputting the visual features and the semantic features into a visual modality variational autoencoder and a semantic modality variational autoencoder respectively to generate a first pseudo visual feature and a pseudo semantic feature;

[0014] S140: Inputting the first pseudo visual feature and the pseudo semantic feature into a preconfigured generator, and fusing them to generate a second pseudo visual feature;

[0015] S150: Perform back propagation optimization parameters according to the loss function of the image augmentation model until the overall loss function converges, save the model parameters, and obtain a trained image augmentation model.

[0016] As an example, the loss function includes an adversarial loss function, and the adversarial loss function acquisition step includes:

[0017] Configure visual feature discriminator and semantic feature discriminator;

[0018] Inputting the visual feature and the second pseudo visual feature into a visual feature discriminator to obtain first discriminant information;

[0019] Inputting the semantic feature and the pseudo-semantic feature into a semantic feature discriminator to obtain second discriminant information;

[0020] Determine the adversarial loss function according to the first discriminant information and the second discriminant information, and use the Adam gradient descent algorithm to update the parameters of the visual feature discriminator and the semantic feature discriminator;

[0021] The loss function also includes the total loss function L of the variational autoencoder VAE , reconstruction loss of visual modality variational autoencoder And KL divergence loss, reconstruction loss of semantic modality variational autoencoder and KL divergence loss.

[0022] As an example, in step S110:

[0023] Extracting visual features of the visible class training image using a visual feature extraction model, wherein the visual feature extraction model uses a convolutional neural network and a Transformer encoder as a feature extraction network;

[0024] Input the visible class training image into the convolutional neural network to obtain the feature map;

[0025] The feature map is divided into multi-dimensional feature vector blocks, and each feature vector block is mapped into a one-dimensional vector by linear mapping to obtain multiple feature vectors;

[0026] The feature vector is positionally encoded and embedded into the Transformer encoder, the encoder blocks are repeatedly stacked L times in the encoder, a second-dimensional feature vector is output, and the second-dimensional feature vector is reorganized into a visual feature of a predetermined size.

[0027] As an example, in step S110:

[0028] A semantic feature extraction model is used to extract the semantic features of the visible class training image, and a continuous bag-of-words model obtained through unsupervised training in a text corpus is used as a semantic feature extraction model. The semantic feature extraction model is used to extract the semantic feature vector of the visible class image, and then the semantic feature vector is converted into a semantic feature of a predetermined size through a dimensionality transformation network.

[0029] As an example, the visual modality variational autoencoder in step S120 includes an encoder network E1 and a decoder network D1, wherein the encoder network E1 is a fully convolutional network including n layers of convolution, and the number of filter channels increases layer by layer, for learning deep features; the output of the last convolutional layer in the fully convolutional network is two n-dimensional vectors, namely a mean vector and a variance vector;

[0030] The encoder network E1 maps the visual features to an interval vector represented by a probability distribution N(μ,Σ), and samples to obtain a latent variable Z1, where μ is the mean vector and Σ is the variance vector; the probability distribution of the latent variable Z1 is:

[0031] q1(Z1|x)=N(Z1|μ1,Σ1),p(Z1)=N(Z1|0,I)

[0032] Among them, q1(Z1|x) represents the probability distribution obeyed by the latent variable Z1, p(Z1) represents the prior distribution of the latent variable Z1, which is the unit Gaussian distribution here, μ1 and Σ1 represent the mean and variance of the latent variable Z1, and N represents the normal distribution.

[0033] It should be noted that the encoder part of the variational autoencoder maps the feature data to another latent variable space. This latent space is a statistical distribution function, and its parameters are the mean and variance. In theory, neural networks can fit any function, so the encoder network here maps the input features to mean and variance vectors through a fully convolutional neural network. The variational autoencoder randomly samples an element from the latent space composed of the distribution of the mean and variance, and decodes this element to the original input.

[0034] As an example, the semantic modality variational autoencoder includes an encoder network E2 and a decoder network D2. Both the encoder network E2 and the decoder network D2 use two fully connected layers for encoding and decoding. The semantic feature is input into the encoder network E2 to obtain the probability distribution of the latent variable Z2. The decoder network D2 restores the probability distribution of the latent variable Z2 to the approximate probability distribution of the original data, that is, generates a pseudo-semantic feature similar to the semantic feature. And update the parameters e2 and d2 of the encoder network E2 and the decoder network D2, and calculate the reconstruction loss:

[0035] q2(Z2|a)=N(Z2|μ2,Σ2),p(Z2)=N(Z2|0,I)

[0036]

[0037]

[0038] Among them, q2(Z2|a) represents the probability distribution obeyed by the latent variable Z2, p(Z2) represents the prior distribution of the latent variable Z2, which is a unit Gaussian distribution, μ2 and Σ2 represent the mean and variance of the latent variable Z2, and N represents the normal distribution. represents the reconstruction loss of the semantic modality variational autoencoder, σ represents the logical operation, Denotes the L2 norm squared.

[0039] As an example, in step S130, the total loss function L of the visual modality variational autoencoder and the semantic modality variational autoencoder is calculated. VAE , the total loss function L VAE It consists of the total reconstruction loss and KL divergence loss of the visual modality autoencoder and the semantic modality variational autoencoder. The total reconstruction loss is used to calculate the similarity between the second pseudo visual feature and the visual feature data, as shown in the following formula:

[0040]

[0041] Among them, L VAE is the sum of the loss of the image visual modality and the semantic modality variational autoencoder, represents the reconstruction loss of the visual modality variational autoencoder, represents the reconstruction loss of the semantic modality variational autoencoder, q1(Z1|x) and q2(Z2|a) represent the probability distributions of the latent variables Z1 and Z2, p(Z1) and p(Z2) represent the prior distributions of the latent variables Z1 and Z2, respectively. KL is the KL divergence loss, represents the weight of the reconstruction loss term, which is used to reduce the difference between the generated features and the real features. β represents the weight of the KL divergence loss term, which is used to encourage the network to learn a wider distribution. Given that the dimension of the latent variable space is n, the KL divergence loss is defined as:

[0042]

[0043] Among them, L KL represents KL divergence loss; μ i represents the mean of the spatial dimension i; ∑ i represents the variance of the spatial dimension i.

[0044] As an example, the generator network in step S140 is a multilayer perceptron network containing two fully connected hidden layers; the first pseudo visual feature is globally averaged pooled to obtain a pseudo visual feature of a predetermined size, which is input into the generator network with the pseudo semantic feature to generate a new feature, the generated feature is transformed using the reshape function to obtain a second pseudo visual feature, and the parameters of the generator are updated using the Adam gradient descent algorithm, and the generator loss function is as follows:

[0045]

[0046] in, Represents the first pseudo-visual feature and pseudo-semantic features The second pseudo visual feature generated after the generator G.

[0047] As an example, the adversarial loss function of the visual feature discriminator D and the semantic feature discriminator D* is as follows:

[0048]

[0049]

[0050] Among them, L D represents the adversarial loss function of the visual feature discriminator D, represents the adversarial loss function of the semantic feature discriminator D*, E x represents the expectation of visual feature x, Represents the first pseudo-visual feature and pseudo-semantic features The synthetic expectation, E a represents the expectation of semantic feature a, Representing pseudo-semantic features expectations, represents the visual feature discriminator of the input visual feature x, Represents the first pseudo visual feature of the input and pseudo-semantic features The second pseudo visual feature generated, Represents the input second pseudo visual feature Visual feature discriminator

[0051] The second aspect of the present invention provides an image classification method based on an image augmentation model of a variational autoencoder and an adversarial generative network

[0052] Input the visual features and semantic features of the unseen training images into the image augmentation model as described above to generate pseudo visual features of the unseen training images;

[0053] The generated pseudo visual features are used to train an image classifier, and the visual features of the unseen test images to be identified are input into the classifier for classification to obtain the classification results.

[0054] A third aspect of the present invention provides an electronic device, comprising a processor, an input device, an output device and a memory, wherein the processor, input device, output device and memory are connected in sequence, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the above method.

[0055] A fourth aspect of the present invention provides a readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method as described above.

[0056] 3. Beneficial effects

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] (1) The example of the present invention aims at the problem of zero-shot image classification. It combines the advantages of variational autoencoders and generative adversarial networks, fully integrates the visual information and semantic information of the image to generate more effective samples, and proposes an image augmentation model training method based on variational autoencoders and generative adversarial networks that effectively synthesizes the visual features of unseen images. The method combines a variational autoencoder with a stable training process with a generative adversarial network with clear generated samples. The visual features and semantic features of the image sample are input at the same time to effectively match the visual information and semantic information of the image, improve the quality of the generated data, and effectively solve the problem of missing unseen images in zero-shot learning. The generated pseudo samples are used to train the classifier, and the zero-shot learning is converted into classical supervised learning, thereby improving the accuracy of zero-shot image classification.

[0059] (2) The example of the present invention encodes and decodes features of different modes through a variational autoencoder. The encoder part constructs a unique probability distribution for each sample and samples it. The decoder reconstructs the data to generate sample features that are closer to the real data distribution and have higher robustness.

[0060] (3) The example of the present invention inputs the outputs of the visual and semantic modality variational autoencoders in series into the generator network, fully integrating the visual information and semantic information of the image, and can more effectively mine the association between visual features and semantic features, thereby more effectively synthesizing visual features and semantic features, and reducing the impact of imbalanced generated data on the model; and the example of the present invention effectively combines the advantages of the variational autoencoder and the generative adversarial network, using the variational autoencoder to directly compare the difference between the generated data and the original data through the encoding and decoding steps, and the advantages of the stable training process and the advantage of the adversarial generative network to generate clear pictures, thereby improving the stability and discrimination ability of the generative model.

[0061] (4) The example of the present invention constructs a network model with two-modal variational autoencoders, a generative adversarial network generator and two discriminator networks, which effectively synthesizes visual features and semantic features that are closer to the real data distribution, and ensures the alignment relationship between different modalities to synthesize more effective visual features; in addition, a multi-module loss function including a variational autoencoder loss function, a generator loss function, and a visual and semantic feature discriminator adversarial loss function is constructed, and the variational autoencoder adversarial generative network model is trained based on the multi-module loss function, which effectively alleviates the problems of gradient explosion and model collapse, and improves the performance of the model.

[0062] (5) The present invention uses a variational autoencoder to generate pseudo samples of unseen categories against the generative network model, which effectively solves the problem of missing samples of unseen categories, and uses the generated pseudo samples to train a classifier to classify unseen test images, thereby increasing the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required to be used in the embodiments are briefly introduced below.

[0064] Figure 1 It is a flow chart of an image augmentation model training method based on a variational autoencoder and a generative adversarial network according to an embodiment of the present invention;

[0065] Figure 2 is a schematic diagram of the structure of an image augmentation model according to an embodiment of the present invention;

[0066] Figure 3 is a network structure diagram of a visual feature extraction model in an embodiment of the present invention;

[0067] Figure 4 is a network structure diagram of a semantic feature extraction model in an embodiment of the present invention;

[0068] Figure 5 is a network structure diagram of a visual modality variational autoencoder in an embodiment of the present invention;

[0069] Figure 6 is a network structure diagram of a semantic modality variational autoencoder in an embodiment of the present invention;

[0070] Figure 7 is a network structure diagram of a generator G in an embodiment of the present invention;

[0071] Figure 8 : is a network structure diagram of a visual feature discriminator D in an embodiment of the present invention;

[0072] Fig. 9 : is a network structure diagram of a semantic feature discriminator D* in an embodiment of the present invention;

[0073] Fig.10 It is a flowchart of using the image augmentation model of an embodiment of the present invention for zero-sample image classification. DETAILED DESCRIPTION

[0074] In order to enable technicians in this technical field to better understand the technical solution of the present invention, the following is a detailed description of an image augmentation model training method and an image classification method based on a variational autoencoder and a generative adversarial network of the present invention in conjunction with the accompanying drawings.

[0075] like Figure 1 and Figure 2As shown, this example provides an image augmentation model training method based on a variational autoencoder and a generative adversarial network, the method comprising the following steps:

[0076] S110: Obtain a visible class training image, and extract visual features and semantic features of the visible class training image.

[0077] Specifically, in this example, the visible class training images refer to samples with labels in the feature space, and the unlabeled samples in the feature space are unseen classes. The visual features of the visible class training images are obtained through a pre-configured visual feature extraction model, wherein the visual feature extraction model is obtained by training a convolutional neural network and a Transformer encoder. The semantic features of the visible class training images are obtained through a pre-configured semantic feature extraction model, wherein the semantic feature extraction model is a continuous bag of words model obtained through unsupervised training. The visual features and semantic features of the visible class training images are defined as x and a, respectively, and the visual features and semantic features of the unseen class training images are defined as x and a, respectively. ut 、a ut , the visual features of the unseen test image are defined as x t .

[0078] It should be noted that this example uses AWA2 as the image dataset, which contains text and picture files. The text records the animal species included in the dataset, a total of 50 species, and the attribute labels of each category. The picture files include 37,322 pictures of 50 categories of animals, 30,337 pictures of 40 categories in the training set, and 6,985 pictures of 10 categories in the test set. This example dataset is selected from AWA2, in which the number of category attributes is 75, the number of visible classes is 40, the number of unseen classes is 10, the number of visible class samples is 23,337, and the number of unseen class samples is 7,265. It should be understood that other datasets can be selected here, and the limitation of the present invention should not be understood here.

[0079] like Figure 3 As shown, in one embodiment, the visual feature extraction module is composed of a VGGNet16 model and a Transformer encoder. The image is input into the convolutional neural network VGGNet16 to output a feature map of size 16×16×1024. The feature map is then divided into 256 1024-dimensional feature vector blocks, and each block is then mapped to a one-dimensional vector through linear mapping. 256 vectors of length 512 (generally referred to as tokens) are obtained through mapping. The vectors are positionally encoded and embedded in the Transformer encoder. The encoder blocks are repeatedly stacked L times in the Transformer encoder, and 256 512-dimensional feature vectors are output, which are reorganized into visual features of visible class images with a size of 16×16×512. The value of L in this example is 5.

[0080] like Figure 4 As shown, in one embodiment, the semantic feature extraction model is obtained through unsupervised training. First, a continuous bag-of-word model (CBOW) obtained through unsupervised training in a large-scale text corpus is obtained in advance; the category semantic label information of the visible class image is input into the above model to obtain its semantic feature vector, and then a dimensionality transformation network with only one hidden layer is used to convert the semantic feature vector into a semantic feature with a dimension size of 1×1×512.

[0081] S120: Preconfigure an image augmentation model, where the image augmentation model includes a visual modality variational autoencoder, a semantic modality variational autoencoder, and a generator configured according to a generative adversarial network.

[0082] Specifically, the image augmentation model in this example is pre-configured, wherein the visual modality variational autoencoder includes an encoder network E1 and a decoder network D1, and the semantic modality variational autoencoder includes an encoder network E2 and a decoder network D2.

[0083] like Figure 5 As shown in Figure 1, the encoder network E1 is a fully convolutional network (FCN) containing n layers of convolutions. The number of filter channels increases layer by layer to learn deep features. The output of the last convolution in the fully convolutional network is two n-dimensional vectors: the mean vector and the variance vector.

[0084] The encoder network E1 maps the visual features to an interval vector represented by a probability distribution N(μ,Σ), and samples to obtain a latent variable Z1, where μ is the mean vector and Σ is the variance vector. The mean and variance contain the structural information of the input features. The probability distribution of the latent variable Z1 is:

[0085] q1(Z1|x)=N(Z1|μ1,Σ1),p(Z1)=N(Z1|0,I) (1)

[0086] Among them, q1(Z1|x) represents the probability distribution that the latent variable Z1 obeys, p(Z1) represents the prior distribution of the latent variable Z1, which is a unit Gaussian distribution here, μ1 and Σ1 represent the mean and variance of the latent variable Z1, and N represents the normal distribution. The encoder network E1 in this example is a 3-layer convolution, with filter sizes of 32, 64, and 128 respectively. The last convolution outputs two two-dimensional vectors. The pseudo-visual features finally output by the decoder network have the same dimension as the real visual features, that is, 16×16×512. Among them, the filters of the convolution layer are used to identify certain specific features of the image. Each filter will slide the feature map of the previous layer of convolution. The shallow layer of the convolutional neural network generally detects primary features such as "edge" and "color". As the number of convolutional layers increases, the filter convolves the features to obtain various new features. When it comes to the deep layer, the filter can extract deeper features.

[0087] Further considering that the structures of the decoder network D1 and the encoder network E1 are basically symmetrical, since the image feature dimension in the encoder is reduced after passing through the full convolutional network, the upsampling layer is used in the decoder to slowly increase the image feature dimension, while the number of filter channels is gradually reduced, and the image feature dimension is gradually integrated to the original feature dimension size. The decoder network D1 generates the first pseudo visual feature similar to the original visual feature of the image And update the parameters e1 and d1 of encoder E1 and decoder D1, and calculate the reconstruction loss of decoder D1:

[0088]

[0089]

[0090] Among them, the visual feature x is obtained by convolution operation of Transformer feature and semantic feature, that is, x = t*a. represents the reconstruction loss of the visual modality variational autoencoder, Denotes the L2 norm squared.

[0091] like Figure 6 As shown in FIG. 1 , the semantic modal variational autoencoder includes an encoder network E2 and a decoder network D2, both of which use two fully connected layers for encoding and decoding. The semantic features are input into the encoder network E2 to obtain the probability distribution of the latent variable Z2. The decoder network D2 restores the probability distribution of the latent variable to the approximate probability distribution of the original data, that is, generates a pseudo-semantic feature similar to the original semantic feature. And update the parameters e2 and d2 of the encoder network E2 and the decoder network D2, and calculate the reconstruction loss:

[0092] q2(Z2|a)=N(Z2|μ2,Σ2),p(Z2)=N(Z2|0,I) (4)

[0093]

[0094]

[0095] Among them, q2(Z2|a) represents the probability distribution obeyed by the latent variable Z2, p(Z2) represents the prior distribution of the latent variable Z2, which is a unit Gaussian distribution, μ2 and Σ2 represent the mean and variance of the latent variable Z2, and N represents the normal distribution. represents the reconstruction loss of the semantic modality variational autoencoder, σ represents the logical operation, represents the L2 norm square;

[0096] S130: Inputting the visual features and semantic features into a visual modality variational autoencoder and a semantic modality variational autoencoder respectively to generate a first pseudo visual feature and a pseudo semantic feature.

[0097] Specifically, the visual features obtained in step S110 are input into the visual modality variational autoencoder for encoding to obtain the first pseudo visual features; the semantic features obtained in step S110 are input into the semantic modality variational autoencoder for encoding to obtain the pseudo semantic features. The size of the pseudo semantic features finally output in this example is 1×1×512, and the pseudo visual features finally output have the same dimension as the real visual features, i.e., 16×16×512.

[0098] S140: Input the first pseudo visual feature and the pseudo semantic feature into a preconfigured generator, and fuse them to generate a second pseudo visual feature.

[0099] Specifically, in this example, the generator network is a multi-layer perceptron (MLP) network with two fully connected hidden layers, and the input of the network is the first pseudo visual feature and pseudo-semantic features The first pseudo visual feature is globally averaged pooled to obtain a pseudo visual feature of a predetermined size, whose feature size is 1×1×C; the pseudo visual feature and the pseudo semantic feature are input into the generator network to generate a new feature, and the generated feature is transformed into a second pseudo visual feature of size H×W×C using the reshape function, and the parameters of the generator are updated using the Adam gradient descent algorithm, where the generator loss function is as follows:

[0100]

[0101] in, Represents the first pseudo-visual feature and pseudo-semantic features The second pseudo visual feature generated after the generator G;

[0102] In this example, the first pseudo visual feature and pseudo semantic feature containing visual information and semantic information respectively are input into the generator network in series. After fully integrating the image visual information and semantic information, a second pseudo visual feature of size 16×16×512 is generated. This feature may be either a visible class or an unseen class.

[0103] S150: Perform back propagation optimization parameters according to each loss function of the image augmentation model until the overall loss function converges, save the model parameters, and obtain a trained image augmentation model.

[0104] Specifically, the loss functions of the image augmentation model in this example include an adversarial loss function, a visual modality variational autoencoder loss function, a semantic modality variational autoencoder loss function, and a generator loss function.

[0105] In one embodiment, the total loss function L VAE It consists of the total reconstruction loss and KL (Kullback–Leibler) divergence loss of the visual modality autoencoder and the semantic modality variational autoencoder. The total reconstruction loss is used to calculate the similarity between the generated feature data and the original feature data, while the KL divergence loss is used to test the difference between the mean μ and the variance Σ, as shown in the following formula:

[0106]

[0107] Among them, L VAE is the sum of the losses of the image visual modality autoencoder and the semantic modality variational autoencoder, represents the reconstruction loss of the visual modality variational autoencoder, represents the reconstruction loss of the semantic modality variational autoencoder, q1(Z1|x) and q2(Z2|a) represent the probability distributions of the latent variables Z1 and Z2, p(Z1) and p(Z2) represent the prior distributions of the latent variables Z1 and Z2, respectively. KL is the KL divergence loss, represents the weight of the reconstruction loss term, which is used to reduce the difference between the generated features and the real features. β represents the weight of the KL divergence loss term, which is used to encourage the network to learn a wider distribution.

[0108] Given a latent variable space dimension of n, the KL divergence loss is defined as:

[0109]

[0110] Among them, L KL represents KL divergence loss; μ i represents the mean of the spatial dimension i; ∑ i represents the variance of the spatial dimension i.

[0111] In one embodiment, the steps of obtaining the adversarial loss function are as follows:

[0112] First, a visual feature discriminator D and a semantic feature discriminator D* are constructed, and the visual features of the visible class image and the second pseudo-visual features in the above step are input into the discriminator D to discriminate true or false, and obtain the first discriminant information;

[0113] The semantic features and pseudo-semantic features of the above-mentioned visible class images are input into the discriminator D* to distinguish true from false, and the second discriminant information is obtained. The adversarial loss function is determined according to the first discriminant information and the second discriminant information respectively.

[0114] Specific as Figure 8 and 9 As shown, in this example, the visual feature discriminator D includes a fully connected group and a Sigmoid binary classifier, and the semantic feature discriminator D* includes a fully connected hidden layer and a Sigmoid binary classifier. The final outputs of the visual feature discriminator D and the semantic feature discriminator D* are 0 and 1 values, 0 indicates that the feature is false, and 1 indicates that the feature is true. The Adam gradient descent algorithm is used to update the parameters of the discriminator. The adversarial loss function of the visual feature discriminator D and the semantic feature discriminator D* is as follows:

[0115]

[0116]

[0117] Among them, L D represents the adversarial loss function of the visual feature discriminator D, L D * represents the adversarial loss function of the semantic feature discriminator D*, E x represents the expectation of the image visual feature x, Represents the first pseudo-visual feature and pseudo-semantic features The synthetic expectation, E a represents the expectation of the image semantic feature a, Representing pseudo-semantic features expectations, represents the visual feature discriminator of the input visual feature x, Represents the input pseudo visual features and pseudo-semantic features The second pseudo visual feature generated, Represents the input second pseudo visual feature Visual feature discriminator.

[0118] Repeat the above steps, use the visual features and semantic features of the visible class training images as the input of the image augmentation model, and train the model by back propagation based on each loss function. Continuously update and optimize the parameters in the visual modality variational autoencoder, semantic modality variational autoencoder, generator, visual features, and semantic feature discriminators until the overall loss function converges, and obtain the trained variational autoencoder adversarial generative network model. Save the model parameters and complete the image augmentation model training. In addition, in this example, when the model is trained, the number of images processed per batch (batch_size) is 32, the Adam optimizer is used, the learning rate (learning_rate) is set to 0.0001, the activation function is ReLU, the Dropout discard rate is 0.5, and the maximum number of training rounds is 100000.

[0119] This example combines the advantages of variational autoencoders and generative adversarial networks, fully integrates the visual information and semantic information of the image to generate more effective samples, can effectively match the visual features and semantic features of the image, improve the quality of generated data, and effectively solve the problem of missing unseen images in zero-shot learning. The generated pseudo samples are used to train the classifier, transforming zero-shot learning into classic supervised learning, thereby improving the accuracy of zero-shot image classification.

[0120] This example also provides an image classification method based on variational autoencoder and adversarial generative network, including the following steps:

[0121] S210: Inputting the visual features and semantic features of the unseen training images into the image augmentation model as described above to generate pseudo visual features of the unseen training images.

[0122] Specifically, Fig.10 As shown, the visual features x of the unseen training images are ut and semantic feature a ut Input the trained image augmentation model to generate unseen pseudo visual features

[0123] S220: Using the generated pseudo visual features to train an image classifier, inputting the visual features of the unseen test image to be identified into the classifier for classification, and obtaining a classification result.

[0124] Specifically, the Softmax classifier is trained using the pseudo visual features of the unseen class and the class labels, and the visual features x t Input the trained classifier to obtain its classification accuracy.

[0125] In this example, we use unseen pseudo-visual features and category labels to train a Softmax classifier for zero-shot image classification. The training set of the Softmax classifier is is the input of a given classifier, m is the number of training samples, t i is the number of the class to which the sample belongs, t i ∈{1,2,…,C}, C is the total number of image categories, and the Softmax classifier is defined as

[0126]

[0127] in, Indicates that given input , the probability that the input data belongs to category k, k = 1, ..., C, is the parameter of the classifier. The output of the classifier is a column vector with C rows and 1 column. Each row represents the probability that the current input is identified as category k. The sum of all row elements is 1. is the parameter to be estimated for the Softmax classifier, and constitutes the parameter matrix

[0128] This example uses the Softmax classifier to classify unseen images for any feature vector. The k categories corresponding to the maximum probability are selected as the classification results of the current image, and the classification results are compared with the marked true results. If the results are consistent, the classification is correct. If the results are inconsistent, the classification is wrong.

[0129] The gradient descent method is used to optimize the model parameters. The loss function of the Softmax classifier is:

[0130]

[0131] in, Solved using the conjugate algorithm Unconstrained optimization problem is used as an evaluation indicator for test classification accuracy. The visual features x of the unseen test image are t Input the trained Softmax classifier and take the classification label corresponding to the value with the highest probability output by the classifier as the prediction result of the classifier.

Claims

1. A method for training an image augmentation model based on a variational autoencoder and a generative adversarial network, characterized in that: The method comprises the following steps: S110: Obtain a visible class training image, and extract visual features and semantic features of the visible class training image; S120: pre-configuring an image augmentation model, wherein the image augmentation model includes a visual modality variational autoencoder, a semantic modality variational autoencoder, and a generator configured according to a generative adversarial network; S130: Inputting the visual features and the semantic features into a visual modality variational autoencoder and a semantic modality variational autoencoder respectively to generate a first pseudo visual feature and a pseudo semantic feature; S140: Inputting the first pseudo visual feature and the pseudo semantic feature into a preconfigured generator, and fusing them to generate a second pseudo visual feature; S150: Perform back propagation optimization parameters according to the loss function of the image augmentation model until the overall loss function converges, save the model parameters, and obtain a trained image augmentation model.

2. The image augmentation model training method based on variational autoencoder and adversarial generative network according to claim 1, characterized in that: The loss function includes an adversarial loss function, and the adversarial loss function acquisition step includes: Configure visual feature discriminator and semantic feature discriminator; Inputting the visual feature and the second pseudo visual feature into a visual feature discriminator to obtain first discriminant information; Inputting the semantic feature and the pseudo-semantic feature into a semantic feature discriminator to obtain second discriminant information; Determine the adversarial loss function according to the first discriminant information and the second discriminant information, and use the Adam gradient descent algorithm to update the parameters of the visual feature discriminator and the semantic feature discriminator; The loss function also includes the total loss function L of the variational autoencoder VAE , reconstruction loss of visual modality variational autoencoder And KL divergence loss, reconstruction loss of semantic modality variational autoencoder and KL divergence loss.

3. The image augmentation model training method based on variational autoencoder and adversarial generative network according to claim 1, characterized in that: In step S110: Extracting visual features of the visible class training image using a visual feature extraction model, wherein the visual feature extraction model uses a convolutional neural network and a Transformer encoder as a feature extraction network; Input the visible class training image into the convolutional neural network to obtain the feature map; The feature map is divided into multi-dimensional feature vector blocks, and each feature vector block is mapped into a one-dimensional vector by linear mapping to obtain multiple feature vectors; The feature vector is positionally encoded and embedded into the Transformer encoder, the encoder blocks are repeatedly stacked L times in the encoder, a second-dimensional feature vector is output, and the second-dimensional feature vector is reorganized into a visual feature of a predetermined size.

4. The image augmentation model training method based on variational autoencoder and adversarial generative network according to claim 3, characterized in that: In step S110: A semantic feature extraction model is used to extract the semantic features of the visible class training image, and a continuous bag-of-words model obtained through unsupervised training in a text corpus is used as a semantic feature extraction model. The semantic feature extraction model is used to extract the semantic feature vector of the visible class image, and then the semantic feature vector is converted into a semantic feature of a predetermined size through a dimensionality transformation network.

5. The image augmentation model training method based on variational autoencoder and adversarial generative network according to claim 1, characterized in that: In step S120, the visual modality variational autoencoder includes an encoder network E1 and a decoder network D1, wherein the encoder network E1 is a fully convolutional network including n layers of convolutions, and the number of filter channels increases layer by layer, so as to learn deep features; the output of the last convolutional layer in the fully convolutional network is two n-dimensional vectors, namely a mean vector and a variance vector; The encoder network E1 maps the visual features to an interval vector represented by a probability distribution N(μ,Σ), and samples to obtain a latent variable Z1, where μ is the mean vector and Σ is the variance vector; the probability distribution of the latent variable Z1 is: q1(Z1|x)=N(Z1|μ1,Σ1),p(Z1)=N(Z1|0,I) Among them, q1(Z1|x) represents the probability distribution obeyed by the latent variable Z1, p(Z1) represents the prior distribution of the latent variable Z1, which is the unit Gaussian distribution here, μ1 and Σ1 represent the mean and variance of the latent variable Z1, and N represents the normal distribution.

6. The image augmentation model training method based on variational autoencoder and adversarial generative network according to claim 1, characterized in that: The semantic modality variational autoencoder includes an encoder network E2 and a decoder network D2. Both the encoder network E2 and the decoder network D2 use two fully connected layers for encoding and decoding. The semantic features are input into the encoder network E2 to obtain the probability distribution of the latent variable Z2. The decoder network D2 restores the probability distribution of the latent variable Z2 to the approximate probability distribution of the original data, that is, generates a pseudo-semantic feature similar to the semantic feature. And update the parameters e2 and d2 of the encoder network E2 and the decoder network D2, and calculate the reconstruction loss: q2(Z2|a)=N(Z2|μ2,Σ2),p(Z2)=N(Z2|0,I) Among them, q2(Z2|a) represents the probability distribution obeyed by the latent variable Z2, p(Z2) represents the prior distribution of the latent variable Z2, which is a unit Gaussian distribution, μ2 and Σ2 represent the mean and variance of the latent variable Z2, and N represents the normal distribution. represents the reconstruction loss of the semantic modality variational autoencoder, σ represents the logical operation, Denotes the L2 norm squared.

7. The image augmentation model training method based on variational autoencoder and adversarial generative network according to claim 5, characterized in that: In step S130, the total loss function L of the visual modality variational autoencoder and the semantic modality variational autoencoder is calculated. VAE , the total loss function L VAE It consists of the total reconstruction loss and KL divergence loss of the visual modality autoencoder and the semantic modality variational autoencoder. The total reconstruction loss is used to calculate the similarity between the second pseudo visual feature and the visual feature data, as shown in the following formula: Among them, L VAE is the sum of the loss of the image visual modality and the semantic modality variational autoencoder, represents the reconstruction loss of the visual modality variational autoencoder, represents the reconstruction loss of the semantic modality variational autoencoder, q1(Z1|x) and q2(Z2|a) represent the probability distributions of the latent variables Z1 and Z2, p(Z1) and p(Z2) represent the prior distributions of the latent variables Z1 and Z2, respectively. KL is the KL divergence loss, represents the weight of the reconstruction loss term, which is used to reduce the difference between the generated features and the real features. β represents the weight of the KL divergence loss term, which is used to encourage the network to learn a wider distribution. Given that the dimension of the latent variable space is n, the KL divergence loss is defined as: Among them, L KL represents KL divergence loss; μ i represents the mean of the spatial dimension i; ∑ i represents the variance of the spatial dimension i.

8. The image augmentation model training method based on variational autoencoder and adversarial generative network according to claim 1, characterized in that: In step S140, the generator network is a multilayer perceptron network containing two fully connected hidden layers; the first pseudo visual feature is globally averaged pooled to obtain a pseudo visual feature of a predetermined size, which is input into the generator network with the pseudo semantic feature to generate a new feature, the generated feature is transformed using a reshape function to obtain a second pseudo visual feature, and the parameters of the generator are updated using the Adam gradient descent algorithm, and the generator loss function is as follows: in, Represents the first pseudo-visual feature and pseudo-semantic features The second pseudo visual feature generated after the generator G.

9. The image augmentation model training method based on variational autoencoder and adversarial generative network according to claim 3, characterized in that: The adversarial loss function of the visual feature discriminator D and the semantic feature discriminator D* is as follows: Among them, L D represents the adversarial loss function of the visual feature discriminator D, represents the adversarial loss function of the semantic feature discriminator D*, E x represents the expectation of visual feature x, Represents the first pseudo-visual feature and pseudo-semantic features The synthetic expectation, E a represents the expectation of semantic feature a, Representing pseudo-semantic features expectations, represents the visual feature discriminator of the input visual feature x, Represents the first pseudo visual feature of the input and pseudo-semantic features The second pseudo visual feature generated, Represents the input second pseudo visual feature Visual feature discriminator.

10. An image classification method based on an image augmentation model of variational autoencoder and adversarial generative network, Inputting the visual features and semantic features of the unseen training images into the image augmentation model according to any one of claims 1 to 8 to generate pseudo visual features of the unseen training images; The generated pseudo visual features are used to train an image classifier, and the visual features of the unseen test images to be identified are input into the classifier for classification to obtain the classification results.

Images