Small sample classification method based on multi-modal linear programming

Abstract

A kind of small sample classification method based on multi-modal linear programming, by constructing and training the visual-to-semantic autoencoder used to predict the semantic features of unlabeled samples, after the cluster center of unlabeled sample is mapped to semantic space, by linear programming model, use class semantic features as reference semantic features in semantic space, assign label to unlabeled sample cluster center to match initial prototype and cluster center and obtain revised class prototype, after multi-modal feature combination, based on the improved optimization strategy of alternating least squares (ALS), the feature vectors and weight matrix of support set samples and query set samples are alternately optimized, iterate until the relative residual of prediction score no longer changes, obtain the result of reducing feature representation deviation and classifier weight deviation.The present application predicts the semantic features of query samples through the autoencoder, solves the information imbalance problem of support samples and query samples, reduces the deviation of class representation through the multi-modal feature correction mechanism, and optimizes the feature representation and classifier weight by alternating optimization.

Description

Technical Field

[0001] This invention relates to a technique in the field of image classification, specifically a few-sample classification method based on multimodal linear programming. Background Technology

[0002] Few-sample image classification is an important research area in computer vision. This method aims to bridge the gap between machine learning and human learning. However, most existing methods only use unimodal features. Due to the extreme scarcity of sample data, unimodal visual features cannot provide sufficient information, while semantic modalities can provide additional prior knowledge, thereby improving classification accuracy. Summary of the Invention

[0003] This invention addresses the problems of existing technologies failing to eliminate cross-class bias in features extracted from new class samples and the inaccuracy of class distribution learned from single-modal information based on a small number of samples. It proposes a small-sample classification method based on multimodal linear programming, which uses an autoencoder to predict the semantic features of query samples, thereby solving the information imbalance problem between support samples and query samples. It reduces the bias in class representation through a multimodal feature correction mechanism and uses an alternating optimization approach to optimize feature representation and classifier weights.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a few-sample classification method based on multimodal linear programming. It constructs and trains a vision-to-semantic autoencoder to predict the semantic features of unlabeled samples. After mapping the cluster centers of unlabeled samples to the semantic space, a linear programming model is used in the semantic space with category semantic features as baseline semantic features. Labels are assigned to the cluster centers of unlabeled samples to match the initial prototype and the cluster centers, resulting in a corrected category prototype. After multimodal feature combination, an improved optimization strategy based on alternating least squares (ALS) is used to alternately optimize the feature vectors and weight matrices of the support set samples and the query set samples. The iteration continues until the relative residuals of the predicted scores no longer change, resulting in a reduction of feature representation bias and classifier weight bias.

[0006] The autoencoder is trained in the following way: first, the feature extractor F is pre-trained using training set samples. θ The features of the new class samples in the test set are used to extract the features of the new class samples in the test set. Then, N classes are randomly selected from the test set, and K+L samples are randomly selected from each class to construct a few-sample task containing a support set and a query set. The image features and class semantic features of the support set are input into the autoencoder for training, where K is the number of samples of each class in the support set and L is the number of samples of each class in the query set.

[0007] The aforementioned baseline semantic features are extracted from category labels using the language model GloVe.

[0008] The cluster centers of the unlabeled samples are obtained by clustering the sample features of the query set in the small sample task.

[0009] The initial prototype was obtained by averaging the sample features of each class in the support set of a small sample task.

[0010] The aforementioned correction refers to: matching the initial prototype with the cluster centers, and then weighting and summing the paired cluster centers and the initial prototype to obtain the corrected category prototype.

[0011] The multimodal feature combination refers to concatenating the visual and semantic features of all samples in the current small sample task. The semantic features of the support set samples are the label word vectors of the corresponding categories, i.e., category semantic features. The semantic features of the query set samples are predicted by the trained autoencoder based on the visual features of the query set.

[0012] The training set, test set, support set, and query set are obtained in the following ways:

[0013] 1) Use publicly available benchmark datasets as experimental data. By dividing the benchmark dataset into two parts, training set and test set, the data is divided into two parts: training set and test set. For example, miniImageNet has a total of 100 categories. Data from 80 categories is used as the training set, and samples from the remaining 20 categories are used as the test set.

[0014] 2) Randomly select N categories in the test set, and randomly select K+L samples from each category to construct an N-way K-shot task, where N*K samples are used as the support set and the remaining N*L samples are used as the query set.

[0015] The improved optimization strategy based on alternating least squares refers to: constructing a linear classifier, deriving a closed-form solution for features and weights based on the alternating least squares (ALS) algorithm, iteratively optimizing features and weights, and synchronously updating the prediction results until the relative residuals of the prediction results no longer change.

[0016] This invention relates to a system for implementing the above-described method, comprising: a residual neural network unit, an autoencoder unit, a category prototype correction unit, and an alternating optimization unit, wherein: the residual neural network unit is pre-trained based on information from a large number of labeled samples in the training set to obtain a feature extractor model; the autoencoder unit performs image feature to semantic feature mapping and semantic feature to visual feature reconstruction based on visual and semantic feature sample pairs in the support set to obtain a semantic feature prediction model; the category prototype correction unit performs feature clustering and cross-modal label assignment based on query set sample information, and performs weighted summation of paired cluster centers and initial category prototypes to obtain more representative category prototypes; the alternating optimization unit performs alternating optimization of weights and feature representations based on feature information of multimodal combinations in the task to obtain debiased features and weights. Technical effect

[0017] This invention utilizes an autoencoder to complete the missing modalities of unlabeled samples, performs cross-modal category prototype correction, and alternately optimizes features and weights. By introducing multimodal information, an alternating optimization mechanism is introduced in the optimization of features and classifier weights. Compared with existing technologies, the model obtained by this invention achieves advanced classification accuracy, and the distinguishability of feature representations and the representativeness of category prototypes are significantly better than existing technologies. Attached Figure Description

[0018] Figure 1 This is a flowchart of the present invention;

[0019] Figure 2 This is a diagram illustrating small sample classification.

[0020] Figure 3 A schematic diagram of multimodal cluster center label assignment;

[0021] Figure 4 This is a schematic diagram illustrating the alternating optimization of features and weights;

[0022] Figure 5 The example shows the distribution of intra-class and inter-class distances of features before and after alternating optimization. Detailed Implementation

[0023] like Figure 1 and Figure 2 As shown, this embodiment relates to a small-sample classification method based on multimodal linear programming, including:

[0024] Step 1) First, train the autoencoder using the support set data, specifically including:

[0025] Step 1.1) Calculate the word vectors of the category labels using the GloVe model, as the category semantic features;

[0026] Step 1.2) Construct an autoencoder, train it using image features and semantic features from the input support set, and calculate the mean squared error (MSE) as the loss function in both the semantic space and the image feature space, specifically: in: Let represent the supporting dataset for the task, u represent the baseline semantic features of the category, and x represent the visual features of the image. g(·|φ) and g(·|φ) represent the encoder and decoder modules in the autoencoder model, respectively, and λ represents the weighting coefficient between the mapping loss and the reconstruction loss.

[0027] Step 2) as Figure 3 As shown, the modification of category prototype features and the expansion of multimodal information specifically include:

[0028] Step 2.1) Use the K-means clustering algorithm to cluster the image features of the query set samples to obtain N cluster centers c. j ,j=1,...,N.

[0029] Step 2.2) Use the trained autoencoder to map the cluster centers to the semantic space to obtain the semantic cluster centers.

[0030] Step 2.3) Using the category semantic features as the baseline semantic features, calculate the Euclidean distance between each semantic cluster center and the baseline semantic features. in: Let u represent the i-th semantic cluster center. j This represents the baseline semantic feature of the j-th category.

[0031] Step 2.4) Construct a 0-1 programming model for paired cluster centers. Compared with the baseline semantic features u j The total distance between ∈G is minimized to calculate the globally optimal decision variable o. ij Specifically: minimize∑ (i,j)∈C× G d ij o ij ,subject to∑ j∈G o ij for i∈C, ∑ i∈G o ij for j∈G o ij ∈{0,1}for ,j∈C,G, where: O ij Let C represent the decision variables, and G represent the semantic cluster center and the baseline semantic feature set, respectively.

[0032] Step 2.5) Based on the optimal solution of the decision variables, assign classification labels to the cluster centers and match them with the initial prototype p of the corresponding categories. n By averaging, we obtain the corrected category prototype. Where n = 1, ..., N represents the category label.

[0033] Step 2.6) Predict the semantic features of the query set samples. in: and These represent the support set and the query set, respectively. i This is the i-th query sample.

[0034] Step 3) as Figure 4 As shown, alternating optimization of features and classifier parameters achieves the effect of bias removal in sample features and classifier weights, thereby improving classification accuracy. This engineering technique specifically includes:

[0035] Step 3.1) Construct a linear classifier by concatenating the corrected category prototypes and category semantic features as the initial weights W = [w1; ...; w N ], Where: N represents the number of categories in the task, u n Represents the baseline semantic features of the nth category. This represents the modified category prototype, and (·||·) represents the feature concatenation operation.

[0036] Step 3.2) Concatenate the image features and semantic features of the support set samples. And concatenate the image features in the query set with the predicted semantic features. The features of the two sets are combined into a single feature matrix. in: The pseudo-semantic features representing the query sample are obtained by inputting the visual features of the query sample into an autoencoder for prediction. and These represent the multimodal features of the support set and the query set, respectively.

[0037] Step 3.3) Predict the initial probability vector using the initial features of the query set and the initial weights of the classifier. The initial probability vector of the support set is The initial probability matrix of all samples in the current task is obtained as follows

[0038] Step 3.4) Based on the Alternating Least Squares (ALS) algorithm, construct an optimization model for the features and weights. Where: W represents the classifier weight, X represents the feature to be optimized, and λ1 and λ2 both represent weight coefficients.

[0039] Step 3.5) By taking the derivative of the objective function with respect to the features X and weights W respectively, the closed-form solutions for the features and weights can be derived: Where: I represents the identity matrix, Y represents the prediction probability matrix, and the subscript t represents the number of iterations.

[0040] Step 3.6) Iteratively optimize the features and weights Y t =W t X t The prediction results are updated synchronously until the relative residuals no longer change: r t =||Y t -W t X t || F , Where: r t Represents the relative residual, where ε is a very small positive number, typically set to 1e. -4 .

[0041] Step 4) Classify the image based on the trained classifier and optimized features.

[0042] Through specific practical experiments, we performed few-shot classification evaluations on three benchmark datasets: MiniImageNet, TieredImageNet, and CIFAR-FS, using a 5-way 1 / 5-shot method. The 15-query parameter constructs a few-shot classification task to run the above method. Here, 5-way means 5 classes per task, 1 / 5-shot means 1 or 5 training samples per class, and 15-query means 15 test samples per class. The experimental data obtained are as follows: with ResNet12 and WRN-28-10 backbone networks, the accuracy achieved on MiniImageNet, TieredImageNet, and CIFAR-FS datasets in 1-shot and 5-shot settings is as follows: ResNet12: 78.14% / 85.34%, WRN-28-10: 79.02% / 86.35%; ResNet12: 83.60% / 86.72%, WRN-28-10: 84.79% / 89.96%; ResNet12: 83.42% / 87.23%, WRN-28-10: 88.48% / 89.39%.

[0043] The evaluation metric was calculated under two experimental settings: 5-way 1-shot and 5-way 5-shot. 600 test tasks were constructed for each setting, and the accuracy was calculated on these test tasks. The average accuracy across all test tasks was then used as the algorithm's performance evaluation. The task construction was as follows: Five classes were randomly selected from the test dataset, and 16 samples were randomly selected from each class (for the 5-way 1-shot task), with 15 samples as the query set and 1 sample as the support set. For the 5-way 5-shot case, 20 samples were selected for each class, with 15 samples as the query set and 5 samples as the support set. The experimental prediction results are as follows:

[0044] Table 1. Classification accuracy performance of different few-shot classification algorithms on MiniImageNet, TieredImageNet, and CIFAR-FS datasets. Accuracy is obtained by calculating the average classification accuracy across 600 test tasks. ResNet12 and WRN-28-10 represent feature extraction networks. The gray-labeled rows represent the results of our method, while the others represent the results of existing comparative methods.

[0045] like Figure 5 As shown, the distribution of intra-class and inter-class distances of features before and after alternating optimization is illustrated. The results show that the feature vectors after alternating optimization exhibit better intra-class aggregation and inter-class separation characteristics, thus being more discriminative.

[0046] Compared with existing technologies, this method uses cross-modal class prototype correction and alternating optimization of features and classifier weights, which greatly improves the accuracy of the model. Results show that this invention achieves near-state results on MiniImageNet, TieredImageNet, and CIFAR-FS datasets, and the optimized features are more discriminative.

[0047] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A small-sample classification method based on multimodal linear programming, characterized in that, By constructing and training a vision-to-semantic autoencoder to predict the semantic features of unlabeled samples, the cluster centers of unlabeled samples are mapped to the semantic space. Then, a linear programming model is used to assign labels to the cluster centers of unlabeled samples to match the initial prototype and the cluster centers, and to obtain the corrected category prototype. After multimodal feature combination, an improved optimization strategy based on alternating least squares is used to alternately optimize the feature vectors and weight matrices of the support set samples and the query set samples. The iteration continues until the relative residual of the prediction score no longer changes, resulting in a reduction of feature representation bias and classifier weight bias. The small sample classification method specifically includes: Step 1) First, train the autoencoder using the support set data, specifically including: Step 1.1) Calculate the word vectors of the category labels using the GloVe model, as the category semantic features; Step 1.2) Construct an autoencoder, train it using image features and semantic features from the input support set, and calculate the mean squared error (MSE) as the loss function in both the semantic space and the image feature space, specifically: ,in: This represents the supporting dataset in the task. The baseline semantic features representing the category, For the visual features of an image, and These represent the encoder and decoder modules in the autoencoder model, respectively. This represents the weighting coefficients between the mapping loss and the reconstruction loss; Step 2) Correcting the category prototype features and expanding multimodal information, specifically including: Step 2.1) Use the K-means clustering algorithm to cluster the image features of the query set samples to obtain N cluster centers. ; Step 2.2) Use the trained autoencoder to map the cluster centers to the semantic space to obtain semantic cluster centers. ; Step 2.3) Using the category semantic features as the baseline semantic features, calculate the Euclidean distance between each semantic cluster center and the baseline semantic features. ,in: Represents the i-th semantic cluster center. Represents the baseline semantic features of the j-th category; Step 2.4) Construct a 0-1 programming model for paired cluster centers. Compared with the baseline semantic features The global optimal decision variable is calculated by minimizing the total distance between them. Specifically: , , ,in: Let C and G represent the decision variables, respectively, and let C and G represent the semantic cluster center and the baseline semantic feature set. Step 2.5) Based on the optimal solution of the decision variables, assign classification labels to the cluster centers and match them with the initial prototypes of the corresponding categories. By averaging, we obtain the corrected category prototype. ,in: Indicates category label; Step 2.6) Predict the semantic features of the query set samples. ,in: and These represent the support set and the query set, respectively. This is the i-th query sample; Step 3) Alternately optimize the features and classifier parameters to obtain the bias-reducing effect of sample features and classifier weights, thereby improving the engineering technology effect of classification accuracy. Specifically, this includes: Step 3.1) Construct a linear classifier by concatenating the corrected class prototypes and class semantic features as the initial weights of the classifier. Where: N represents the number of categories in the task, Represents the baseline semantic features of the nth category. This represents the modified category prototype. This indicates the operation of feature concatenation; Step 3.2) Concatenate the image features and semantic features of the support set samples. And concatenating the image features in the query set with the predicted semantic features, The features of both sets are then combined into a single feature matrix. ,in: The pseudo-semantic features representing the query sample are obtained by inputting the visual features of the query sample into an autoencoder for prediction. and These represent the multimodal features of the support set and the query set, respectively. Step 3.3) Predict the initial probability vector using the initial features of the query set and the initial weights of the classifier. The initial probability vector of the support set is The initial probability matrix for all samples in the current task is obtained as follows: ; Step 3.4) Based on the Alternating Least Squares (ALS) algorithm, construct an optimization model for the features and weights. ,in: Represents the classifier weights. Indicates the features to be optimized. and All represent weighting coefficients; Step 3.5) By taking the derivative of the objective function with respect to the features X and weights W respectively, the closed-form solutions for the features and weights can be derived: , ,in: Let Y represent the identity matrix, Y represent the prediction probability matrix, and the subscript t represent the number of iterations; Step 3.6) Iteratively optimize features and weights. The prediction results are updated synchronously until the relative residuals no longer change. ,in: Indicates relative residuals. To represent a very small positive number, it is generally set to... ; Step 4) Classify the image based on the trained classifier and optimized features.

2. The small-sample classification method based on multimodal linear programming according to claim 1, characterized in that, The autoencoder is trained in the following way: first, the feature extractor is pre-trained using training set samples. This is used to extract features from new class samples in the test set; N categories are randomly selected from the test set, and K+L samples are randomly selected from each category to construct a few-sample task containing a support set and a query set. The image features and category semantic features of the support set are input into the autoencoder for training, where K is the number of samples in each category of the support set and L is the number of samples in each category of the query set.

3. The small-sample classification method based on multimodal linear programming according to claim 1, characterized in that, The aforementioned baseline semantic features are extracted from category labels using the language model GloVe.

4. The small-sample classification method based on multimodal linear programming according to claim 1, characterized in that, The cluster centers of the unlabeled samples are obtained by clustering the sample features of the query set in the small sample task.

5. The small-sample classification method based on multimodal linear programming according to claim 1, characterized in that, The initial prototype was obtained by averaging the sample features of each class in the support set of a small sample task.

6. The small-sample classification method based on multimodal linear programming according to claim 1, characterized in that, The aforementioned correction refers to matching the initial prototype with the cluster centers, and then weighting and summing the paired cluster centers and the initial prototype to obtain the corrected category prototype.

7. The small-sample classification method based on multimodal linear programming according to claim 1, characterized in that, The multimodal feature combination refers to concatenating the visual and semantic features of all samples in the current small sample task. The semantic features of the support set samples are the label word vectors of the corresponding categories, i.e., category semantic features. The semantic features of the query set samples are predicted by the trained autoencoder based on the visual features of the query set.

8. The small-sample classification method based on multimodal linear programming according to claim 1, characterized in that, The improved optimization strategy based on alternating least squares refers to: constructing a linear classifier, deriving a closed-form solution for features and weights based on the alternating least squares algorithm, iteratively optimizing features and weights, and synchronously updating the prediction results until the relative residuals of the prediction results no longer change.

9. A small-sample classification system based on multimodal linear programming that implements the method of any one of claims 1-8, characterized in that, include: The system comprises a residual neural network unit, an autoencoder unit, a category prototype correction unit, and an alternating optimization unit. The residual neural network unit is pre-trained based on information from a large number of labeled samples in the training set to obtain a feature extractor model. The autoencoder unit performs image feature mapping to semantic features and reconstruction from semantic features to visual features based on visual and semantic feature sample pairs in the support set to obtain a semantic feature prediction model. The category prototype correction unit performs feature clustering and cross-modal label assignment based on the query set sample information, and performs a weighted sum of the paired cluster centers and the initial category prototype to obtain a more representative category prototype; the alternating optimization unit performs alternating optimization of weights and feature representations based on the feature information of multimodal combinations in the task to obtain debiased features and weights.

Images