A rebalancing long-tail image data classification method based on multi-modal data

Abstract

The application belongs to the field of image classification, and designs a long-tail image data classification method based on multi-modal data rebalancing. The method realizes the application of image-text multi-modal learning in long-tail image classification problem, aims to use relatively balanced and easy-to-obtain, richly expandable text data to supervise the learning of image features of the model, and improves the classification effect of the model on all categories through two-stage training. In the first stage, the image and text encoders in the CLIP large-scale pre-training model are used, the correlation of the two modal data is established through the contrast learning method, the mutual information of the intra-class image and text is enhanced, and the inter-class difference is enlarged. In the second stage, the image and text encoders are frozen, and a multi-layer perceptron is added after the image encoder, and the class balanced sampling strategy and the rebalancing loss function are used to train a small number of cycles, and the classification ability of the model for the tail classes is further improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image classification, and particularly relates to a long-tail image data classification method based on multi-modal data. BACKGROUND

[0002] Image classification is a basic problem in the field of computer vision, aiming to distinguish different categories of images according to the semantic information of the images and achieve the minimum classification error. Deep learning performs well in image classification tasks to a large extent due to large amounts of high-quality training data, in which the number of samples contained in different categories is the same, avoiding the negative effects of unbalanced training samples. However, the data collected in real applications usually presents a long-tail distribution, and it is difficult for the model to learn good feature representations of all categories.

[0003] The problem of long-tail data imbalance in classification is essentially an imbalance problem of the final classifier weights. Currently, the mainstream uses information enhancement to solve this problem. This method aims to introduce additional information to assist during model training, thereby improving model performance. Head-tail knowledge transfer uses the within-class variance knowledge in the head classes to guide the feature enhancement of the tail classes, so that the features of the tail classes have greater within-class variance. Model pre-training uses a contrast learning method to first perform self-supervised learning for pre-training, and then performs normal training on the long-tail data. Knowledge distillation uses the output of a well-trained teacher model to guide the training of a student model. Self-supervised training uses labeled samples to train a supervised model, then uses the model to generate pseudo-labels for unlabeled samples, and finally uses labeled samples and unlabeled samples to train the model again.

[0004] Thanks to the additional introduction of knowledge assistance, these methods of transferring or enhancing data improve the classification effect of the tail classes without causing additional negative effects on the head classes, thus fundamentally solving the problem of lack of sufficient tail class information in long-tail data. It is a direction worth exploring. However, simply using data enhancement techniques often cannot effectively divide the head classes and tail classes. The head classes have more samples and will be subjected to more enhancement processing, thereby further strengthening the information imbalance phenomenon. SUMMARY

[0005] In view of the deficiencies of the prior art, the application designs a long-tail image data classification method based on multi-modal data rebalancing.

[0006] A long-tail image data classification method based on multi-modal data rebalancing, the specific steps are as follows:

[0007] Step 1: Preprocess the image data and text data;

[0008] Given a mini-batch of image data I = {I1,..., IN} and corresponding label text data T = {T1,..., TN}, where N is the batch size; represent the image and label text data of class i in the mini-batch as N and N respectively. and where and are subsets of I and T with size n;

[0009] Step 2: Dimensionality reduction encoding is performed on the image modal data and label text modal data obtained in step 1.

[0010] For any , the label text modal data is fitted into the prompt template "a photo of a {class}" to become a sentence and the token is calculated.

[0011] The image in and the sentence token are sent into the image encoder E I and the text encoder E T respectively for calculation to obtain the image modal embedding representation and the label description modal embedding representation

[0012]

[0013] where D is the output dimension after alignment of the two modal Embeddings.

[0014] Step 3: Similarity matching of image modal data and label text modal data is calculated.

[0015] According to the two modal embedding representations obtained in step 2, it is judged whether the image modal data and the label text modal data match by the cosine similarity S.

[0016]

[0017] where, is the image modal embedding representation belonging to class j, is the label description modal embedding representation belonging to class k, and represents the Hadamard product operator, T represents the transpose of the matrix.

[0018] Step 4: Pre-training the image modal embedding representation and the label description modal embedding representation through contrastive learning to establish the intra-class relevance between image modal data and label text modal data, while expanding the inter-class similarity boundary, as the first stage, i.e., pre-training the CLIP model;

[0019] Step 4.1: The number of positive samples in a mini-batch is n 2 , the similarity between all images of the same class modal data and label text modal data, and the number of negative samples is N 2 -n 2 , the similarity between i-class image modal data and other different classes of label text modal data, the cosine similarity matrix of the mini-batch

[0020] Step 4.2: For any image modal embedding representation and label description modal embedding representation, encode the position index of the corresponding modal data in the mini-batch as 1 if it is of the same class, and set the position index of different classes as 0 to obtain a two-modal data encoding matrix of the mini-batch

[0021] Step 4.3: Calculate the loss function of the first stage of contrastive learning:

[0022]

[0023] Where τ is the temperature coefficient in contrastive learning, set to 0-0.1 initially, and updated during the training process; S i,j is the cosine similarity between i-class image modal embedding representation and j-class label description modal embedding representation; S i,k is the cosine similarity between i-class image modal embedding representation and k-class label description modal embedding representation;

[0024] Step 4.4: Knowledge distillation is performed on the excellent feature extraction capability of the CLIP pre-training model, and a distillation loss function is used to assist in completing knowledge transfer to avoid overfitting of the CLIP pre-training model during training:

[0025]

[0026] Where S' is the cosine similarity calculated by the original CLIP pre-training model after freezing for the same data;

[0027] Step 5: Calculate the final loss of the first stage:

[0028]

[0029] where a is a hyper-parameter, used to adjust the proportion of original CLIP model knowledge distillation in model pre-training.

[0030] Step 6: Repeat steps 2-5 to update the image encoder parameters by using the gradient descent algorithm for backpropagation, to realize the first stage CLIP model pre-training.

[0031] Step 7: For any given mini-batch image modal data of size N The sentence token of all category label text modal data of category number C is After calculation by image encoder and text encoder respectively, the embedding representation is obtained and D is the output dimension after alignment of the two modal Embedding.

[0032] Step 8: Calculate the original cosine similarity of image embedding representation f I and label description embedding representation f T :

[0033] S ori =f I ⊙(f T ) T

[0034] Where ⊙ represents the Hadamard product operator, and T represents the transpose of the matrix.

[0035] The obtained represents the predicted value of each image category based on f I and f T of the CLIP model after the first stage training.

[0036] Step 9: Because the image modal data presents a long-tail distribution, using image modal data and text modal data pairs for matching classification still cannot get rid of the imbalance problem, therefore, the CLIP model is re-balanced to change the sensitivity of image embedding representation f I to label description embedding representation f T , as the second stage, the specific steps are as follows:

[0037] Step 9.1: After the MLP mapping, the dimension of f I is unchanged, and then the similarity with f T is calculated:

[0038] S mlp =MLP(f I )⊙(f T ) T

[0039] wherein, ⊙ represents the Hadamard product operator, T represents the transpose of the matrix;

[0040] Step 9.2: Add the cosine similarity in the above formula to the category number weight to obtain the balanced cosine similarity:

[0041]

[0042] wherein, represents the proportion of the number of the jth category of samples in the total number;

[0043] Step 9.3: Train the formula (4) and the one-hot label of the modal data using the cross-entropy loss function to calculate the loss, and then perform back propagation to update the MLP parameters:

[0044]

[0045] wherein, τ is the temperature coefficient after the first stage of training is frozen;

[0046] Step 10: Calculate the formula (5) and the formula (6) to obtain the final prediction output: ori

[0047]

[0048] wherein, λ is a hyperparameter, used to adjust the weight of the rebalancing of the MLP module; represent the matching degree of the image modal data to all category description text modal data after calculation, that is, the prediction result, so argmax (S final ) is the final prediction category.

[0049] The present application has the beneficial technical effects

[0050] ​​​​​A long-tail image data classification method based on multi-modal data is realized, and the application of image-text multi-modal learning in long-tail image classification is realized. In image classification, the training data of long-tail distribution can cause the learning process of the model to be easily dominated by the head category with rich sample data, and the learning and modeling ability of the tail category is limited, which brings challenges to the final classification accuracy. The feature learning process of the model and the rebalancing process for the long-tail problem are decoupled as two stages to learn. The first stage keeps the original sampling strategy of the data set unchanged, fully utilizes all the data for image encoder feature learning, and introduces the feature representation of the text mode to provide supervision information. In order to further improve the classification performance of the species with small sample quantity and eliminate the learning deviation caused by the long-tail data to the encoder training, in the second stage, the image and text encoders are frozen, and a multilayer perceptron (MLP) is added after the image encoder to rebalance the image encoder. In order to retain the good feature representation learned in the first stage, the idea of residual connection is adopted, and the similarity of the image-text pair output by the model before and after the addition of the MLP is combined as the final prediction value.

[0051] The present application can not only learn good image features, but also use the text information of the image label to assist in guiding the model classification, and convert the classification prediction problem into an image-text pairing problem by using the contrast learning method, so as to realize the supervision of image classification training by text data.

[0052] Compared with the prior art, the technical scheme of the present application greatly reduces the calculation amount, and the feature learning ability of the model for the long-tail distribution data set is very close to that of the balanced data set. The method is quite sufficient for learning the classification boundary between the image-text pairs of different categories, and can learn the differences between categories from limited sample data, which also reflects the promoting effect of label text description on image feature extraction learning. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The first stage of the CLIP model contrast learning pre-training framework of the present application is shown in the figure;

[0054] Figure 2 The second stage of the CLIP model of the present application is shown in the figure. DETAILED DESCRIPTION

[0055] The present application will be further described below in combination with the drawings and examples;

[0056] This paper, based on the perspective of image-text co-training, explores how to use textual information to fill the information gap caused by the scarcity of tail-category images. First, contrastive learning pre-training is used to associate each image with its labeled textual description, maximizing the mutual information between image-text pairs within a class while also increasing the variance between classes. Next, information about the number of image samples between classes is introduced to account for the long-tail distribution of images. A multi-layer perceptron is then added to retrain the rebalanced image encoder.

[0057] A rebalanced long-tail image data classification method based on multimodal data, the specific steps are as follows:

[0058] Step 1: Preprocess image data and text data;

[0059] Given a mini-batch of image data I={I1,...,I N} and the corresponding label text data T={T1,...,T N}, where N is the batch size; the two modal data of image and label text belonging to class i in the mini-batch are represented as and in and is a subset of I and T, with size n;

[0060] Step 2: Image modality data obtained in step 1 and label text modal data Perform dimensionality reduction encoding;

[0061] For any Put the label text modal data into the prompt template "aphoto ofa{class}" to turn it into a sentence and calculate the token;

[0062] Will The image and sentence token in are fed into the image encoder E I and text encoder E T Perform calculations to obtain the image modality embedding representation and label description modality embedding representation

[0063]

[0064] in D is the output dimension after the two modal embeddings are aligned;

[0065] Step 3: Calculate the similarity matching between image modality data and label text modality data;

[0066] According to the two modal embedding representations obtained in step 2, whether the image modal data and the label text modal data match is judged by cosine similarity S;

[0067]

[0068] wherein, is the image modal embedding representation belonging to the j-th class, is the label description modal embedding representation belonging to the k-th class, and represents the Hadamard product operator, and T represents the transpose of a matrix;

[0069] Step 4: The image modal embedding representation and the label description modal embedding representation are pre-trained by contrastive learning to establish the relevance between the image modal data and the label text modal data within the class, and to expand the inter-class similarity boundary, as the first stage, that is, to pre-train the CLIP model;

[0070] Step 4.1: The number of positive samples in a mini-batch is n 2 is the similarity between all image modal data and label text modal data of the same class, and the number of negative samples is N 2 -n 2 is the similarity between the i-th image modal data and other different class label text modal data, and the cosine similarity matrix of the mini-batch is

[0071] Step 4.2: For any image modal embedding representation and label description modal embedding representation, the position index of the corresponding modal data of the same class in the mini-batch is encoded as 1, and the position index of the different class is set as 0 to obtain a two-modal data encoding matrix of the mini-batch

[0072] Step 4.3: The loss function of the first stage contrastive learning is calculated:

[0073]

[0074] Wherein τ is the temperature coefficient in the contrastive learning, which is set to 0-0.1 initially and is updated with the training process; S i,j is the cosine similarity between the i-th image modal embedding representation and the j-th label description modal embedding representation; S i,k is the cosine similarity between the i-th image modal embedding representation and the k-th label description modal embedding representation;

[0075] Step 4.4: Knowledge distillation is performed on the excellent feature extraction capability of the CLIP pre-training model, and a distillation loss function is used to assist knowledge transfer to avoid overfitting of the CLIP pre-training model during training:

[0076]

[0077] where S' is the cosine similarity calculated by the original CLIP pre-training model after freezing on the same data;

[0078] Step 5: Calculate the final loss of the first stage:

[0079]

[0080] where α is a hyperparameter used to adjust the proportion of CLIP model knowledge distillation in model pre-training;

[0081] Step 6: Repeat steps 2-5 to update the image encoder parameters using the gradient descent algorithm for backpropagation, and implement the first stage CLIP model pre-training;

[0082] Step 7: Given a mini-batch image modal data of size N The sentence token of all category label text modal data of C categories is The embedding representation is calculated after passing through the image encoder and text encoder respectively and D is the output dimension after alignment of the two modal Embedding;

[0083] Step 8: Calculate the original cosine similarity of the image embedding representation f I and the label description embedding representation f T obtained in step 7:

[0084] S ori =f I ⊙(f T ) T

[0085] where ⊙ represents the Hadamard product operator, and T represents the transpose of the matrix;

[0086] Get f I and f T represent the prediction value of each image category based on f I and f T after the first stage training of the CLIP model;

[0087] Step 9: Because the image modal data is long-tailed distribution, using image modal data and text modal data pair for matching classification still cannot get rid of the imbalance problem, therefore, the CLIP model is re-balanced to change the image embedding representation f I The sensitivity of the label description embedding representation f T , as the second stage, the specific steps are as follows:

[0088] Step 9.1: f I After the MLP mapping, the dimension is unchanged, and then f T Calculate the similarity:

[0089] S mlp = MLP(f I )⊙(f T ) T

[0090] Where, ⊙ represents the Hadamard product operator, and T represents the transpose of the matrix;

[0091] Step 9.2: Add the cosine similarity in f The category number weight to get the balanced cosine similarity:

[0092]

[0093] Where, represents the proportion of the number of j-th class samples to the total number;

[0094] Step 9.3: Train f With the one-hot label of modal data using cross-entropy loss function to calculate the loss, and then perform back propagation to update the MLP parameters:

[0095]

[0096] Where τ is the temperature coefficient frozen after the first stage training;

[0097] Step 10: Weighted sum of f And S ori As the final prediction output:

[0098]

[0099] Where λ is a hyperparameter used to adjust the weight of the MLP module re-balancing; Represents the matching degree of this image modal data to all category description text modal data after calculation, that is, the prediction result, so argmax(S final ) is the final prediction category.

[0100] The present application uses the CIFAR100 dataset, the ImageNet2012 dataset and the Places365 dataset in the long-tail classification field. Since the original dataset is evenly distributed, a general processing method is used to divide it into a long-tail distribution dataset. The present application uses Top-1 accuracy as the main evaluation index, that is, the index of the value with the maximum probability in the final classifier output vector is used as the accuracy of the model predicted class.

[0101]

[0102] wherein A t Top-1 accuracy, A u is the empirical reference accuracy, which is the baseline accuracy A v and the balanced accuracy A b is the maximum value. The baseline accuracy A v represents the test set accuracy of the backbone network used by the algorithm after training on the balanced training set using the cross-entropy loss function, and the balanced accuracy A b represents the test set accuracy of the long-tail method after training on the balanced training set.

[0103] The basic network of the present application is the CLIP pre-trained model, and the two Encoders of ResNet-50 and ViT-Base / 16 published by it are used as image encoders, and the text encoder is the Transformer in the GPT-2 after the CLIP pre-training.

[0104] Table 1 shows the Top-1 accuracy of different algorithms on the CIFAR100-LT dataset under 3 unbalanced proportions ρ of the CIFAR100-LT dataset. The comparison algorithms include traditional class rebalancing methods (Focal Loss, LDAM, etc.), information enhancement methods (OLTR, MiSLAS, etc.), and improved model modules (BBN, RIDE, etc.), and the latest algorithms related to image-text multimodal learning in long-tail classification, such as BALLAD.

[0105] Table 1 Top-1 accuracy under different ρ of CIFAR100-LT;

[0106]

[0107] It can be seen that under the three unbalanced proportion data sets, the method all achieves the optimal effect, compared with BALLAD algorithm also using CLIP pre-training model, when p = 10, 50 and 100, the Top-1 accuracy of ResNet-50 image encoder is increased by 6.6%, 5.6% and 4.8% respectively. And the ViT-16 image encoder has relatively excellent feature learning, so the improvement is small, and the Top-1 accuracy is increased by 1.6%, 0.3% and 0.5% respectively. It should be noted that the text encoder is not frozen in the first stage of BALLAD, and the text encoder parameters still need to be learned and updated, so the computational amount of the present application is greatly reduced compared with it.

[0108] Table 2 shows the relative accuracy index A of each algorithm on the ImageNet-LT dataset r The experimental results.

[0109] Table 2 shows the relative accuracy index A of each algorithm on the ImageNet-LT dataset

[0110]

[0111] It can be seen that the relative accuracy of BMLTC is higher than 90%, which shows that the feature learning ability of the model for long-tail distribution data set is very close to that of balanced data set. But the relative accuracy of BALLAD and BMLTC model is not as good as RIDE, which proves that the reason why this kind of transfer algorithm has superior classification performance on long-tail distribution data set is partly due to the use of CLIP pre-training model with powerful image feature extraction ability. And the A of RIDE u is higher than that of other methods, which shows that the reason why the model achieves good classification effect is not limited to the adjustment of the classifier, although it also improves the overall learning generalization ability, but the improvement for long-tail distribution data set is more obvious, so the relative accuracy is also higher. In addition, the A of information enhancement method u is relatively high, which also shows that information enhancement methods can generally improve the feature learning ability of the model, and the method of designing special loss function is more suitable for the training of long-tail distribution data.

[0112] Table 3 shows the Top-1 accuracy (%) of each part of Places-LT

[0113]

[0114] Table 3 shows the comparison results of each algorithm on the Places-LT dataset. Except for BALLAD and BMLTC, other methods use ResNet-152 as the backbone network. It can be seen that since these algorithms are all based on the pre-trained ResNet-152 model for retraining, the classification effect of each algorithm on the Places-LT dataset is not very different. BMLTC improves the accuracy of ResNet-50 and ViT-16 image encoders by 0.5% and 0.6% respectively compared with BALLAD, and has little or even slightly lower improvement than BALLAD on Few-Shot, but has more improvement on Many-Shot, with an increase of 1.2% and 1.4% respectively.

Claims

1. A rebalanced long-tail image data classification method based on multimodal data, characterized in that: The specific steps are as follows: Step 1: Preprocess image data and text data; Given a mini-batch of image data I={I1,...,I N } and the corresponding label text data T={T1,...,T N }, where N is the batch size; the two modal data of image and label text belonging to class i in the mini-batch are represented as and in and is a subset of I and T, with size n; Step 2: Image modality data obtained in step 1 and label text modal data Perform dimensionality reduction encoding; Step 3: Calculate the similarity matching between image modality data and label text modality data; Step 4: Pre-train the CLIP model by performing comparative learning on the image modality embedding representation and the label description modality embedding representation to establish the intra-category correlation between the image modality data and the label text modality data, while expanding the inter-class similarity boundary. This is the first stage. Step 5: Calculate the final first-stage loss; Step 6: Repeat steps 2-5, using the gradient descent algorithm for backpropagation to update the image encoder parameters to complete the first stage of CLIP model pre-training; Step 7: Given a mini-batch image modality data of size N The sentence tokens of all types of labeled text modal data with the number of types C are The embedded representation is obtained after calculation by the image encoder and text encoder respectively and D is the output dimension after the two modal embeddings are aligned; Step 8: Calculate the image embedding representation f obtained in step 7 I and label description embedding representation f T The original cosine similarity S ori ; Step 9: Because image modality data has a long-tail distribution, using image modality data and text modality data for matching classification still cannot get rid of the imbalance problem. Therefore, the CLIP model is rebalanced to change the image embedding representation f I Embedding representation f for label description T As the second stage, we get Step 10: and S ori The weighted sum is taken as the final prediction output.

2. The method for classifying long-tailed image data based on multimodal data according to claim 1, characterized in that: Step 2 is as follows: For any Put the label text modal data into the prompt template "aphoto ofa{class}" to turn it into a sentence and calculate the token; Will The image and sentence token in are fed into the image encoder E I and text encoder E T Perform calculations to obtain the image modality embedding representation and label description modality embedding representation in D is the output dimension after the two modal embeddings are aligned.

3. The method for classifying long-tailed image data based on multimodal data according to claim 1, characterized in that: Step 3 is as follows: Based on the two modal embedding representations obtained in step 2, the cosine similarity S is used to determine whether the image modal data and the label text modal data match; in, is the modality embedding representation of the image belonging to class j, Describes the modal embedding representation for the k-class label, ⊙ represents the Hadamard product operator, and T represents the transpose of the matrix.

4. The method for classifying long-tailed image data based on multimodal data according to claim 1, characterized in that: Step 4 is as follows: Step 4.1: The number of positive samples in a mini-batch is n 2 , is the similarity between all similar image modality data and label text modality data, and the number of negative samples is N 2 -n 2 , is the similarity between the i-type image modality data and other different types of labeled text modality data, the cosine similarity matrix of the mini-batch Step 4.2: For any image modality embedding representation and label description modality embedding representation, encode the position subscript of the corresponding modality data of the same type in the mini-batch as 1, and set the position subscript of the data of different types to 0, to obtain a mini-batch encoding matrix of the two modal data Step 4.3: Calculate the loss function of the first stage contrastive learning: Where τ is the temperature coefficient in contrastive learning, which is set to an initial value of 0 to 0.1 and updated during the training process; S i,j is the cosine similarity between the image modality embedding representation belonging to category i and the label description modality embedding representation belonging to category j; S i,k is the cosine similarity between the embedding representation of the image modality belonging to category i and the embedding representation of the label description modality belonging to category k; Step 4.4: Distill the excellent feature extraction capabilities of the CLIP pre-trained model and use a distillation loss function to assist in knowledge transfer to avoid overfitting the CLIP pre-trained model during training: Among them S ' This is the cosine similarity calculated on the same data after the original CLIP pre-trained model is frozen.

5. The method for classifying long-tailed image data based on multimodal data according to claim 4, characterized in that: Step 5 calculates the final loss of the first stage as follows: Among them, α is a hyperparameter used to adjust the proportion of original CLIP model knowledge distillation in model pre-training.

6. The method for classifying long-tailed image data based on multimodal data according to claim 1, characterized in that: The original cosine similarity in step 8 is specifically: Where ⊙ represents the Hadamard product operator, Represents the transpose of a matrix; get Indicates that the CLIP model after the first stage of training is based on f I and f T , the predicted value for each image category.

7. The method for classifying long-tailed image data based on multimodal data according to claim 1, characterized in that: Step 9 is as follows: Step 9.1: f I After MLP mapping, the dimension remains unchanged and then compared with f T Calculate similarity: Where ⊙ represents the Hadamard product operator and T represents the transpose of the matrix; Step 9.2: The cosine similarity in is added with the weight of the number of categories to obtain a balanced cosine similarity: in, Indicates the proportion of the number of samples in the jth category to the total number; Step 9.3: Training the General The loss is calculated using the cross entropy loss function with the one-hot label of the modal data, and then backpropagation is performed to update the MLP parameters: where τ is the temperature coefficient frozen after the first stage of training.

8. The method for classifying long-tailed image data based on multimodal data according to claim 1, characterized in that: The final prediction output is: Where λ is a hyperparameter used to adjust the weight of the MLP module rebalancing; It represents the degree of matching of the image modality data obtained after calculation to all kinds of description text modality data, that is, it represents the prediction result, so argmax(S final ) is the final predicted category.

Images