Small sample classification method based on multi-modal variational contrastive learning

Abstract

A small sample classification method based on multi-modal variational contrast learning constructs and adopts a support set to train a variational autoencoder in an offline stage, and the support set is amplified through the trained variational autoencoder; then the amplified support set is used to train a multi-modal instance-to-distribution conversion model (I2D); in the online stage, the feature points are converted into distribution through the trained multi-modal instance-to-distribution conversion model, so that more accurate small sample classification is realized. The present application converts the multi-modal features at the instance level into the representation at the distribution level through variational inference learning, reduces the class prototype deviation problem caused by small samples, and improves the distinguishability of sample features.

Description

Technical Field

[0001] This invention relates to a technology in the field of computer vision and pattern recognition, specifically a small sample classification method based on multimodal variational contrastive learning, with a sample size of 1 or 5 samples per class. Background Technology

[0002] Few-shot learning addresses the issue that traditional deep learning methods rely on large amounts of labeled data. Most existing few-shot classification methods are based on unimodal features; however, unimodal features cannot provide sufficient information. Semantic features, on the other hand, can serve as prior information about the category, improving the accuracy of feature-based methods and thus increasing classification accuracy. Summary of the Invention

[0003] This invention addresses the information imbalance problem between existing support samples and query sets, the bias in data distribution estimation based on a few samples, and the inaccuracy of class distribution learned from single-modal information of a small number of samples. It proposes a small-sample classification method based on multimodal variational contrastive learning. This invention transforms instance-level multimodal features into distribution-level representations through variational inference learning, mitigating the class prototype bias problem caused by small samples and improving the discriminability of sample features.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a few-shot classification method based on multimodal variational contrastive learning. In the offline stage, a variational autoencoder is constructed and trained using a support set, and the support set is expanded using the trained variational autoencoder. Then, a multimodal instance-to-distribution (I2D) model is constructed and trained using the expanded support set. In the online stage, the feature points are transformed into distributions using the trained multimodal instance-to-distribution model, thereby achieving more accurate few-shot classification.

[0006] The variational autoencoder includes an encoder unit and a decoder unit, wherein: the encoder unit performs image feature to distribution transformation based on visual feature and semantic feature sample pairs in the support set, and applies a contrast constraint between the mean feature of the distribution and the semantic features to train a semantic feature prediction model; the decoder unit performs feature reconstruction based on features generated by sampling in the latent space to amplify the samples in the support set.

[0007] The multimodal instance-to-distribution transformation model includes: a mean network unit, a variance network unit, and a distribution-based metric unit. The mean network unit and the variance network unit take the concatenated features of the visual and semantic modalities from the amplified support set samples as input to obtain the distribution representation in the latent space, namely the mean feature and the variance feature. The distribution-based metric unit performs KL (Kullback-Leibler Divergence) measurement based on the distribution representation of the categories and the distribution representation of the samples to obtain the classification result of the samples.

[0008] The support set refers to a small number of available training samples for each task. Technical effect

[0009] This invention predicts the semantic features of samples in a query set using a variational autoencoder; it adds effective information to the category and sample data through a multimodal feature representation mechanism; and it reduces feature bias caused by a small number of samples by replacing feature points with a distribution representation of the samples. Compared with existing technologies, this invention improves the representativeness of category features and the discriminability of sample features by completing missing modalities in the test samples and by using a multimodal distribution representation of categories and samples, thereby achieving an advanced level of classification accuracy. Attached Figure Description

[0010] Figure 1 This is a flowchart of the present invention;

[0011] Figure 2 This is a training block diagram of the multimodal variational autoencoder of the present invention;

[0012] Figure 3 This is a flowchart illustrating the implementation of the transformation model from multimodal feature points to distribution based on variational autoencoders and the label prediction of samples in the query set according to the present invention. Detailed Implementation

[0013] like Figure 1 As shown in the figure, this embodiment relates to a few-shot classification method based on multimodal variational contrastive learning, which includes the following steps:

[0014] Step 1) Construct and train a mean network f using the support set data. μ and a variance network f σ Variational autoencoders, specifically including:

[0015] Step 1.1) Input the image features of the support set into the encoder unit (encoder(f) of the variational autoencoder. μ f σ The posterior distribution of the sample in the latent space is obtained. Based on the premise that the posterior distribution follows the standard global prior distribution. Construct the KL loss between the prior and posterior distributions: Where: μ i =f μ (x i ), For category y i The corresponding baseline semantic features.

[0016] Step 1.2) Calculate the word vectors of the category labels using the Global Word Representation Vector Model (Glove), as category semantic features;

[0017] Step 1.3) Construct a contrastive loss between the mean features of the distribution and the semantic features of the categories in the latent space: in: For the support set, μ i Let be the mean feature of the i-th support set sample in the latent space. For category y i semantic features, y i Let τ be the category label of the i-th support set sample, and τ be the temperature coefficient.

[0018] Step 1.4) Sample z from the latent space distribution of the sample. i =μ i +∈σ i ∈N(0,I), the image features are reconstructed using the decoder unit in the variational autoencoder. Then, a contrastive loss is constructed between the reconstructed features and the category prototype: Where: z i f represents the features sampled from the latent space. r For decoder networks, To reconstruct the visual features, For category y i The category prototype, where τ is the temperature coefficient.

[0019] Step 1.5) Summation based on the loss functions from steps 1.1, 1.3, and 1.4 This allows for the training of the variational autoencoder network parameters.

[0020] Step 2) Build and train as follows Figure 3 The diagram shows a multimodal means network. A multimodal variance network The multimodal instance-to-distribution transformation model specifically includes:

[0021] Step 2.1) Calculate the distribution of each class in the latent space of the support set using the variational autoencoder trained in Step 1. This is done by averaging the mean and variance of all samples in each class. in: yes A subset of , containing all samples of category n.

[0022] Step 2.2) Sample L samples from the distribution of each category and use the decoder f of the variational autoencoder trained in Step 1. r The samples obtained from sampling in the latent space are reconstructed into image features to augment the support set samples, specifically: z ni =μ n +εσ n ,∈N(0,I), Where: μ n and σ n These are the mean and variance features of category n, respectively, z ni Features sampled from the distribution of each category, For the reconstructed features, X aug This is the feature set for amplification.

[0023] Step 2.3) After concatenating the visual and semantic features of the amplified support set samples, input the concatenated features into the multimodal instance to the distribution transformation model to generate a distribution. By minimizing the KL loss between this distribution and the standard normal distribution, the mapping function from multimodal feature points to the distribution is learned. Specifically, the concatenated multimodal feature v ni =(x ni ||e n ), KL loss Where: e n For the semantic features of category n, x ni Let v be the visual feature of the i-th sample in category n. ni The spliced ​​multimodal features, and These are the mean and variance in the multimodal distribution representation, respectively.

[0024] Step 2.4) Transform the feature points of the query set samples in the task into a distribution representation: use f from the variational autoencoder trained in Step 1. μ Predict samples in the query set pseudo-semantic features μ q =f μ (x q Then, the visual features and pseudo-semantic features of the samples in the query set are concatenated to obtain the multimodal feature v. q =(xq ||μ q ), Then, use the multimodal instance to distribution transformation model trained in step 2.3 to map the multimodal feature representation and distribution representation of the query set samples: Where: x q To query the visual features of the sample, μ q v represents the pseudo-semantic features of this query sample. q To query the multimodal feature representation of the sample. and These are the mean and variance in the multimodal distribution representation of the query sample, respectively.

[0025] Step 2.5) Calculate the distribution of multimodal features of the support set samples in the latent space, and then calculate the distribution for each class: in: Let be the mean feature of the i-th sample in category n. Let be the variance feature of the i-th sample of category n.

[0026] Step 2.6) Based on the categories and the distribution of samples in the query set, determine the labels of the query set samples by measuring the KL divergence between the sample distribution and the category distribution, thus achieving small sample classification. Specifically: in: and These are the distribution representations of the q-th query sample and the n-th category, respectively.

[0027] Through practical experiments, 600 test tasks were constructed under both 5-way 1-shot and 5-way 5-shot settings. Accuracy was calculated on these test tasks, and the average accuracy across all test tasks was used as the algorithm's performance evaluation. The task construction was as follows: Five classes were randomly selected from the test dataset, and 16 samples were randomly selected from each class (for the 5-way 1-shot task): 15 samples were from the query set and 1 sample was from the support set. For the 5-way 5-shot case, 20 samples were selected from each class: 15 samples were from the query set and 5 samples were from the support set. The experimental prediction results are as follows:

[0028] Table 1. Average accuracy of few-shot classification algorithms on 600 test tasks on the MiniImageNet and TieredImageNet datasets. ResNet12 and WRN-28-10 are feature extraction networks.

[0029] Table 2. Average accuracy of few-shot classification algorithms on 600 test tasks on the CIFAR-FS and CUB datasets. ResNet12 and WRN-28-10 are feature extraction networks.

[0030] During the experimental phase, this invention was compared with other representative few-sample classification algorithms. In the table above, the bolded results represent the optimal results. The results show that this invention achieves near-optimal results on datasets such as MiniImageNet, TieredImageNet, CIFAR-FS, and CUB.

[0031] Compared with existing technologies, this invention uses modal completion and distributed representation of multimodal features, which greatly improves the accuracy of the model. Results show that this invention achieves near-state results on datasets such as MiniImageNet, TieredImageNet, CIFAR-FS, and CUB-200-2011.

[0032] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A few-sample classification method based on multimodal variational contrastive learning, characterized in that, In the offline phase, a variational autoencoder is constructed and trained using a support set, and the support set is expanded using the trained variational autoencoder. Then, a multimodal instance-to-distribution (I2D) model is constructed and trained using the expanded support set. In the online phase, the trained I2D model is used to transform feature points into distributions, achieving more accurate small-sample classification. Specifically, this includes: Step 1) Construct and train a mean network using the support set data. and a variance network Variational autoencoders, specifically including: Step 1.1) Input the image features of the support set into the encoder unit of the variational autoencoder. Obtain the posterior distribution of the sample in the latent space. Based on the fact that the posterior distribution follows the prior distribution of the standard population. Construct the KL loss between the prior and posterior distributions: ,in: , For category Corresponding baseline semantic features; Step 1.2) Calculate the word vectors of the category labels using the Global Word Representation Vector Model (Glove), as category semantic features; Step 1.3) Construct a contrastive loss between the mean features of the distribution and the semantic features of the categories in the latent space: ,in: To support the set, Let be the mean feature of the i-th support set sample in the latent space. For category semantic features Let be the class label of the i-th support set sample. Temperature coefficient; Step 1.4) Sampling from the latent space distribution of the samples Image features are reconstructed using the decoder unit in the variational autoencoder. Then, construct a contrastive loss between the reconstructed features and the class prototype: ,in: These are features sampled from the latent space. For decoder networks, To reconstruct the visual features, For category Category prototype, Temperature coefficient; Step 1.5) Summation based on the loss functions from steps 1.1, 1.3, and 1.4 This allows for the training of the variational autoencoder's network parameters. Step 2) Construct and train a multimodal means network. A multimodal variance network The multimodal instance-to-distribution transformation model specifically includes: Step 2.1) Calculate the distribution of each class in the latent space of the support set using the variational autoencoder trained in Step 1. This is done by averaging the mean and variance of all samples in each class. ,in: yes A subset containing categories All samples; Step 2.2) Sample L samples from the distribution of each category and use the decoder unit of the variational autoencoder trained in Step 1. The samples obtained from sampling in the latent space are reconstructed into image features to augment the support set samples, specifically: , ,in: and These are the mean and variance features of category n, respectively. Features sampled from the distribution of each category, For the features of reconstruction, For the amplified feature set; Step 2.3) After concatenating the visual and semantic features of the amplified support set samples, input the concatenated features into the multimodal instance to the distribution transformation model to generate a distribution. By minimizing the KL loss between this distribution and the standard normal distribution, the mapping function from multimodal feature points to the distribution is learned. Specifically, the concatenated multimodal features... , KL loss ,in: For the semantic features of category n, Let be the visual features of the i-th sample in category n. The spliced ​​multimodal features, and These represent the mean and variance in a multimodal distribution representation, respectively. Step 2.4) Transform the feature points of the query set samples in the task into a distribution representation: using the variational autoencoder trained in Step 1. Predict samples in the query set pseudo-semantic features Then, the visual features and pseudo-semantic features of the samples in the query set are concatenated to obtain multimodal features. Then, use the multimodal instance to distribution transformation model trained in step 2.3 to map the multimodal feature representation distribution of the query set samples: ,in: To query the visual features of the sample, This is the pseudo-semantic feature of the query sample. To query the multimodal feature representation of the sample. and These are the mean and variance in the multimodal distribution representation of the query sample, respectively. Step 2.5) Calculate the distribution of multimodal features of the support set samples in the latent space, and then calculate the distribution for each class: ,in: Let be the mean feature of the i-th sample in category n. Let be the variance feature of the i-th sample in category n; Step 2.6) Based on the categories and the distribution of samples in the query set, determine the labels of the query set samples by measuring the KL divergence between the sample distribution and the category distribution, thus achieving small sample classification. Specifically: ,in: and These are the distribution representations of the q-th query sample and the n-th category, respectively.

Images