Long-Tailed Distribution Visual Classification Method Based on Sample-Aware Distillation

Abstract

A visual classification method for long-tailed distribution based on sample-aware distillation, which relates to the fields of image processing and object detection. A long-tailed distribution dataset is established, the input images are preprocessed, a teacher network model and a student network model are trained, the cross-entropy loss and the feature center stability learning loss are calculated, and the selective knowledge distillation loss is calculated using the selective knowledge distillation module; the three losses are used to optimize and train the student network model. A feature center stability learning module is proposed: the global inter-class feature center is calculated, the local feature center is optimized using the inter-class relationship of the global feature center, the richness and expression ability of the data features of the tail classes are enriched, and the optimized local feature center is used to classify the samples; a selective knowledge distillation module is proposed: the knowledge of the teacher model is distilled to the student with emphasis according to the knowledge correctness and confidence of the teacher model and the student model. It can be used for long-tailed image classification, object detection, etc.

Description

Technical Field

[0001] The present invention relates to the fields of image processing and object detection, especially to critical applications related to safety or health, such as autonomous driving and medical / disease diagnosis. In essence, the data has serious imbalances and the practical problem of long-tail effect. It is a visual classification method for long-tail distribution based on sample-aware distillation. Background Art

[0002] Due to its application prospects and practical application value, the long-tail distribution problem has received extensive attention in recent years, and many excellent algorithms have emerged. These algorithms can be roughly divided into three categories: long-tail distribution learning algorithms based on reweighting / resampling, long-tail distribution learning algorithms based on decoupled representation and classifier, and long-tail distribution learning algorithms based on knowledge transfer. Compared with the other two types of algorithms, the long-tail distribution learning algorithms based on knowledge transfer have received relatively high attention at present. This method achieves good performance even when the data of the tail classes is insufficient by transferring the knowledge of the head or the teacher knowledge that has been learned sufficiently to the student model. Such methods usually also use additional modules in combination to enhance the feature expression ability of the samples of the tail classes.

[0003] Knowledge distillation is widely used in model compression and transfer learning. Among them, self-distillation and mutual-distillation are two very important branches in the field of knowledge distillation. Self-distillation is a learning strategy that mimics the output of the model itself at different training stages, while mutual-distillation uses multiple networks (two or more) to train simultaneously. Each network not only receives supervision from the true value labels during the training process, but also refers to the learning experience of its peer networks to improve the generalization ability. During the whole process, the two networks continuously share learning experiences to achieve mutual learning and common progress.

[0004] Existing visual classification methods for long-tail distribution usually increase the weights or resample the tail classes, but these methods often focus too much on the tail classes, and the methods of reweighting and resampling have poor generalization performance. Summary of the Invention

[0005] The purpose of the present invention is to provide a visual classification method for long-tail distribution based on sample-aware distillation, which can effectively improve the classification performance of the long-tail distribution problem, aiming at the problems such as poor generalization performance of the existing reweighting and resampling methods. It uses a deep learning model to improve the feature expression ability of the samples of the tail classes and focuses on knowledge distillation to improve the classification performance of long-tail distribution images.

[0006] The present invention includes the following steps:

[0007] 1) Establish a long-tail distribution data set, sample the data set as input pictures and perform preprocessing;

[0008] 2) Input the preprocessed image in step 1) into the teacher network. After training the teacher network model, input all samples in the training set into the pre-trained teacher model to obtain the mean of the feature vectors, calculate the global class feature center, and take out the label with the highest confidence in the predicted confidence to obtain the teacher prediction result;

[0009] 3) Input the preprocessed image in step 1) into the student network to train the student network model. The sample passes through the feature encoder to obtain a feature vector. The feature vector enters two branches, and calculates the cross-entropy loss and the feature center stability learning loss respectively; take out the class label corresponding to the highest confidence in the predicted confidence to obtain the student prediction result;

[0010] 4) Use the selective knowledge distillation module to calculate the selective knowledge distillation loss from the student prediction result, the teacher prediction result and the true label;

[0011] 5) Optimize and train the student network model by combining the three losses of the cross-entropy loss, the feature center stability learning loss and the selective knowledge distillation loss.

[0012] In step 1), the long-tailed distribution dataset where N and C represent the total number of image samples and the total number of classes respectively; for this dataset, the number of samples corresponding to different classes is unbalanced. For the c-th class, if the corresponding number of samples is n c , then n min <...<n c <...<n max ; Randomly select the number of pictures of the set batch_size as the input of the network in each batch;

[0013] The preprocessing includes normalizing the input picture, randomly cropping it to a fixed size (p*p), and randomly flipping for data augmentation.

[0014] In step 2), the specific steps of training the teacher network model can be: input the preprocessed image into the teacher network (TeacherNetwork), and the image passes through the feature encoder of the network to obtain a 64-dimensional feature vector f t and the network logic prediction output z t , this logic prediction output z t After passing through the function Normalize to obtain the predicted confidence p t , this confidence p t and the true label y form the cross-entropy loss L CE To constrain the model;

[0015] After the training is over, input all samples in the training set into the pre-trained teacher model to obtain the feature vectors f' of all samplest , the logical prediction output z′ t and the prediction confidence p′ t ; Using the feature vector f′ t to calculate the mean of the feature vectors for each category to obtain the global category feature center Q g ; Extract the label with the highest confidence in the prediction confidence p′ t to obtain the teacher's prediction result y t .

[0016] In step 3), the specific steps of training the student network model can be as follows:

[0017] Input the image into the Student Network, where the model of the student network is the same as that of the teacher network, and the student network and the teacher network do not share parameters; the feature encoder of the student network can obtain a 64-dimensional feature vector f for each batch of images s , this feature vector f s enters two branches; the first branch further obtains the network logical prediction output Z S , and after passing through the function normalization to obtain the prediction confidence p s , this prediction confidence p s and the true label y are used to calculate the cross-entropy loss L CE ; The second branch obtains the local category feature center Q by calculating the category feature mean of the feature vector l , and together with the global category feature center Q g is input into the feature center stability learning module to calculate the feature center stability learning loss L SFCL ; Extract the category label corresponding to the highest confidence in the prediction confidence p s to obtain the student's prediction result y s .

[0018] In step 3), the feature center stability learning module is used to improve the feature expression ability of tail class samples, and the specific steps are as follows:

[0019] (3.1) For the training samples, the feature vector f output by the teacher network for the training set images t , through logical prediction to obtain z t , as shown in Equation (1); z t passes through the softmax function to obtain the prediction confidence p t , as shown in Equation (2); the prediction result y t , is the category label corresponding to the maximum value of all prediction confidences, as shown in Equation (3); Input a batch of image sets into the student network, and the corresponding output is the feature vector fs , the logical prediction output z s and the prediction result y s , the calculation process is as follows:

[0020] z t = logits(f t ), z s = logits(f s ) (1)

[0021] p t = softmax(z t ), p s = softmax(z s ) (2)

[0022] where,

[0023] y t = argmax(p t ), y s = argmax(p s ) (3)

[0024] where, the argmax function represents extracting the index corresponding to the highest confidence in the prediction confidence, that is, the class label;

[0025] (3.2) Using the feature center stability learning module to improve the feature expression ability of tail class samples; mainly including the following steps:

[0026] ⅰ. Using the global class feature center Q g obtained by the teacher model as the input of the feature center stability learning module, and using Q g to calculate the affinity matrix representing the inter-class relationship Furthermore, using the softmax function to normalize the affinity matrix A after removing the diagonal to obtain the normalized affinity matrix

[0027] ⅱ. Using the normalized affinity matrix to perform feature fusion on the current batch of sample feature vectors f batch , that is so as to achieve the purpose of enhancing tail class samples;

[0028] ⅲ. Using the normalized affinity matrix to perform weighted optimization on the local class feature center Q l of the current batch, that is using the method of moving weighted average (EMA), using the optimized local class feature center and the global class feature center Qg Update the local class feature center of the current batch in each batch to obtain the optimized local class feature center Q r , and the calculation process formula is as follows:

[0029]

[0030] ⅳ. Use the generated Q r Reduce the distance between the feature of each sample and its corresponding feature center, making it align with its corresponding feature center. The Manhattan distance is used as the distance metric, that is, the sum of the absolute values of the differences in each dimension of the two vectors, to obtain the loss function of the feature center stable learning module, as shown in the following formula:

[0031] L SFCL = ||θ(σ(Q r ), σ(f batch ), y)||1 (6)

[0032] Among them, || ||1 represents the L1 norm, σ(a) = a / ||a|| represents using the Euclidean distance as the distance metric, that is, the L2 norm of the difference between the two vectors, θ(b, c) represents the cosine similarity between b and c, and f batch represents the sample feature of the current batch;

[0033] In step 4), the selective knowledge distillation module is used to improve the classification performance of the long-tail distribution problem, and the specific steps are as follows:

[0034] (4.1) Knowledge distillation usually uses the Kullback-Leibler divergence (KL) to measure the difference in the distributions obtained after the student model and the teacher model predict the same batch of samples; the specific calculation formula of the Kullback-Leibler divergence (KL) is as follows:

[0035]

[0036] Among them, z s , z t are the prediction distributions of the student model and the teacher model respectively;

[0037] (4.2) The selective knowledge distillation module selects knowledge on the basis of knowledge distillation; randomly sample batch_size samples from each batch and input them into the two models to obtain the student prediction result y s and the teacher prediction result y t , the student prediction result y s , the teacher prediction result y tThe three, i.e., the student prediction result $\hat{y}$, the teacher prediction result $\tilde{y}$, and the true label $y$, are used as the inputs of the selective knowledge distillation module to calculate the selective knowledge distillation loss. The distillation weights in this module consist of three cases, and the formula is They are respectively:

[0038] ⅰ. When the teacher prediction result is different from the true label (the teacher makes a wrong prediction), the distillation weight is set to 0;

[0039] ⅱ. When the teacher prediction result is the same as the student prediction result, the distillation weight is set to $p$ r , where $p$ represents the uncertainty measure of the prediction result, and $p = 1 - y$ t , and $r$ takes 2;

[0040] ⅲ. When the teacher prediction result is consistent with the true label (the teacher makes a correct prediction) and the student prediction result is inconsistent with the true label, the distillation weight is set to 1;

[0041] The total selective knowledge distillation loss is calculated as follows:

[0042]

[0043] where $\tau$ is the temperature parameter, represents the weight ratio corresponding to this category calculated according to the number of samples in different categories;

[0044] By optimizing the selective knowledge distillation loss, the distribution difference between the two models can be effectively reduced selectively. In addition, the student model effectively eliminates the wrong knowledge of the teacher model and improves the classification performance of the student model for the long-tail distribution problem;

[0045] The total loss function is as follows:

[0046] $L$ total $= L$ CE $+ L$ SKD $+ \alpha \cdot L$ SFCL (9)

[0047] where $\alpha$ is the balance coefficient of the loss; Repeat the above steps for the set number of iterations until the training ends.

[0048] In step 5), the above three losses are used for network optimization training. During the entire training process of the student network model, the joint cross-entropy loss $L$ CE , the feature center stability learning loss $L$ SFCL , and the selective knowledge distillation loss $L$ SKD are used for network optimization training. The cross-entropy loss $L$ CE is used to constrain the model, and the feature center stability learning loss $L$ SFCL is used to help enhance the feature expression ability of the tail class samples. The selective knowledge distillation loss $L$ SKDIt is used to help distill knowledge to the student network with emphasis to improve the classification performance of the long-tail distribution problem; after training, the feature expression ability of the student network model on the tail class samples is improved, and the correct knowledge of the teacher network model is learned.

[0049] According to the core idea of mutual distillation, by learning to imitate the prediction distribution of the teacher model, while fully learning the teacher distribution, the incorrect knowledge of the teacher model is selectively removed, making the knowledge learned by the student model more reliable and accurate. Using the idea of mutual distillation, the samples in the same batch pass through two different models, and by selectively fitting each other's feature distributions, the distribution difference between the two is minimized while effectively retaining the most correct information.

[0050] Compared with the prior art, the present invention has the following outstanding advantages:

[0051] 1. The present invention first proposes a long-tail distribution visual classification method based on sample-aware distillation. Considering the serious data imbalance problem existing in the long-tail distribution problem, first, the feature expression ability of the tail class samples is improved through the feature center stability learning module, so that the tail class samples have more generalized expression ability; then, the selective knowledge distillation module distills the knowledge of the teacher model to the student with emphasis according to the knowledge correctness and confidence of the teacher model and the student model, further ensuring the accuracy of the classification result while effectively reducing the distribution difference between the two models.

[0052] 2. The idea of data augmentation is cleverly used. By using the global inter-class relationship (inter-class affinity matrix) to perform feature fusion on the sample features in the batch, the effect of data augmentation for the tail class is achieved. This operation is different from conventional data augmentation and does not require pre-operation on the data set in advance, but is spontaneously completed during the training process.

[0053] 3. Selective knowledge distillation is cleverly utilized, so that all samples from the two models, while learning each other's feature distributions, selectively choose the more accurate knowledge in the teacher model. Brief Description of the Drawings

[0054] Figure 1 is the framework of the long-tail distribution visual classification method based on sample-aware distillation of the present invention. Detailed Embodiment

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the following embodiments will further illustrate the present invention in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. On the contrary, the present invention covers any alternatives, modifications, equivalent methods and solutions made within the spirit and scope of the present invention as defined by the claims. Further, in order to enable the public to have a better understanding of the present invention, in the following detailed description of the present invention, some specific details are described in detail. Those skilled in the art can fully understand the present invention without the description of these details.

[0056] The embodiments of the present invention include the following steps:

[0057] (1) Long-tailed distribution dataset Where N and C respectively represent the total number of image samples and the total number of categories. For this dataset, the number of samples corresponding to different categories is unbalanced. For the c-th category, assuming the corresponding number of samples is n c , then n min <... < n c <... < n max . Each batch randomly selects a set number of picture quantities of batch_size as the input of the network;

[0058] (2) Normalize the input pictures, randomly crop them to a fixed size (p*p), and randomly flip them for data augmentation;

[0059] (3) Pre-train the teacher network model: Input the image into the teacher network (Teacher Network). The image can obtain a 64-dimensional feature vector f through the feature encoder of the network t , and further obtain the network logic prediction output z t . This logic prediction output z t After passing through the function is normalized to obtain the prediction confidence p t . This confidence p t and the true label y form the cross-entropy loss L CE to constrain the model. After the training is completed, all samples in the training set are input into the pre-trained teacher model to obtain the feature vectors f′ t , the logic prediction output z′ t and the prediction confidence p′ t of all samples. Use the feature vector f′ t to calculate the mean of the feature vectors for each category to obtain the global category feature center Q g . Take out the label with the highest confidence in the prediction confidence p′ t to obtain the teacher prediction result y t .

[0060] (4) Training the student network model: Input the images into the Student Network, where the model of the student network is the same as that of the teacher network model, and the student network and the teacher network do not share parameters. The feature encoder of the student network can obtain a 64-dimensional feature vector f for each batch of images. s , and this feature vector f s enters two branches. The first branch further obtains the network logical prediction output z s , and after passing through the function and being normalized, the predicted confidence p is obtained. s , and this predicted confidence p s and the true label y are used to calculate the cross-entropy loss L CE . The second branch obtains the local class feature center Q by calculating the class feature mean of this feature vector, l , and together with the global class feature center Q g is input into the feature center stability learning module to calculate the feature center stability learning loss L SFCL . Take out the class label corresponding to the highest confidence in the predicted confidence p s to obtain the student prediction result y s . The student prediction result y s , the teacher prediction result y t and the true label y are used by the selective knowledge distillation module to calculate the selective knowledge distillation loss L SKD . During the entire training process of the student network model, the combined cross-entropy loss L CE , the feature center stability learning loss L SFCL and the selective knowledge distillation loss L SKD are used for network optimization training. The feature center stability learning loss L SFCL helps to enhance the feature expression ability of tail class samples, and the selective knowledge distillation loss L SKD helps to distill knowledge to the student network in a focused manner, thereby improving the classification performance of the long-tail distribution problem.

[0061] (4.1) For the training samples, after going through step (3) in requirement 1, the feature vector f output by the teacher network for the training set images t , which undergoes logical prediction to obtain z t , as shown in Equation (1). z t passes through the softmax function to obtain the predicted confidence p t , as shown in Equation (2). The prediction result y t, is the class label corresponding to the maximum value of all prediction confidences, as shown in Equation (3). After step (4) in Requirement 1, the input of a batch of image sets into the student network outputs the feature vector f s , the logical prediction output z s and the prediction result y s , and the calculation process is as follows in Equations (1) to (3):

[0062] z t = logits(f t ), z s = logits(f s ) (1)

[0063] p t = softmax(z t ), p s = softmax(z s ) (2)

[0064] Among them,

[0065] y t = argmax(p t ), y s = argmax(p s ) (3)

[0066] Among them, the argmax function represents taking out the index corresponding to the highest confidence in the prediction confidence, that is, the class label.

[0067] (4.2) Use the feature center stability learning module to improve the feature expression ability of tail class samples. It mainly includes the following steps:

[0068] ⅰ. Use the global class feature center Q obtained by the teacher model g as the input of the feature center stability learning module, and use Q g to calculate the affinity matrix representing the inter-class relationship Furthermore, use the softmax function to normalize the affinity matrix A after removing the diagonal to obtain the normalized affinity matrix

[0069] ⅱ. Use the normalized affinity matrix to perform feature fusion on the current batch of sample feature vectors f batch , that is so as to achieve the purpose of enhancing tail class samples;

[0070] ⅲ. Use the normalized affinity matrix to perform on the local class feature center Q of the current batch lWeighted optimization, that is and use the method of Exponential Moving Average (EMA), and utilize the optimized local class feature center and the global class feature center Q g to update the local class feature center of the current batch in each batch, and obtain the optimized local class feature center Q r , and the calculation process formula is as shown in Equations (4) and (5):

[0071]

[0072] ⅳ. Utilize the generated Q r to shorten the distance between the feature of each sample and its corresponding feature center, making it align with its corresponding feature center. Here, the Manhattan distance is used as the distance metric, that is, the sum of the absolute values of the differences of each dimension of the two vectors. Finally, the loss function of the feature center stable learning module is obtained, as shown in Equation (6):

[0073] L SFCL = ||θ(σ(Q r ), σ(f batch ), y)||1 (6)

[0074] where, || ||1 represents the L1 norm, σ(a) = a / ||a|| represents using the Euclidean distance as the distance metric, that is, the L2 norm of the difference between two vectors, θ(b, c) represents the cosine similarity between b and c, and f batch represents the sample feature of the current batch.

[0075] The use of the selective knowledge distillation module further improves the classification performance of the long-tailed distribution problem:

[0076] (4.3) Knowledge distillation usually uses the Kullback-Leibler divergence to measure the difference in the distributions obtained after the student model and the teacher model predict the same batch of samples. The specific calculation formula of the Kullback-Leibler divergence (KL) is as shown in Equation (7):

[0077]

[0078] where z s , z t are the prediction distributions of the student model and the teacher model respectively.

[0079] (4.4) The selective knowledge distillation module selects knowledge on the basis of knowledge distillation. Randomly sample batch_size samples from each batch and input them into the two models to obtain the student prediction result y s and the teacher prediction result y tThe predicted result y of the student s , the predicted result y of the teacher t and the true label y are used as the inputs of the selective knowledge distillation module to calculate the selective knowledge distillation loss. The distillation weights in this module consist of three cases, and the formula is They are respectively:

[0080] ⅰ. When the predicted result of the teacher is different from the true label (the teacher's prediction is wrong), the distillation weight is set to 0;

[0081] ⅱ. When the predicted result of the teacher is the same as the predicted result of the student, the distillation weight is set to p r , where p represents the uncertainty measure of the predicted result, and p = 1 - y t , and r takes 2;

[0082] ⅲ. When the predicted result of the teacher is consistent with the true label (the teacher's prediction is correct) and the predicted result of the student is inconsistent with the true label, the distillation weight is set to 1.

[0083] The total selective knowledge distillation loss is calculated as in Equation (8):

[0084]

[0085] where τ is the temperature parameter, represents the weight ratio corresponding to this category calculated according to the number of samples in different categories.

[0086] By optimizing the selective knowledge distillation loss, the distribution difference between the two models can be effectively reduced selectively. In addition, the student model effectively eliminates the wrong knowledge of the teacher model, so as to further improve the classification performance of the student model for the long-tail distribution problem.

[0087] The total loss function is calculated as in Equation (9), where α is the loss balance coefficient. Repeat the above steps for the set number of iterations until the training ends.

[0088] L total = L CE + L SKD + α·L SFCL (9)

[0089] (5) After the above training, the feature expression ability of the student model on the tail class samples is improved, and the correct knowledge of the teacher model is learned. In the test stage, use this student model to predict the categories of the test data set and calculate the classification of the samples;

[0090] (6) Calculate the Top-K (K = 1), the classification accuracy of each category and the overall mean average precision (mAP) according to the classification and the classification evaluation metrics.

[0091] The design feature center stability learning module of the present invention: First, calculate the global inter-class feature center. Secondly, use the inter-class relationship of the global feature center to optimize (refine) the local feature center, thereby enriching the richness and expression ability of the tail class data features. Finally, use the optimized local feature center to classify the samples. The present invention proposes a selective knowledge distillation module: Distill the knowledge of the teacher model to the student model selectively according to the knowledge correctness and confidence of the teacher model and the student model. The present invention can be used for long-tailed image classification (for example, medical / disease diagnosis), object detection (for example, autonomous driving), etc.

[0092] See Figure 1 , the framework of the present invention is:

[0093] Step 1, obtain the input image of the model.

[0094] Sample the data set, and randomly select the number of pictures of the set batch_size as the input of the network for each batch.

[0095] Normalize the input pictures. Randomly crop the CIFAR dataset to the specified size (32*32), randomly crop the ImageNet dataset to the specified size (224*224), and randomly crop to the fixed size (p*p), and randomly flip for data augmentation;

[0096] Step 2, obtain the feature vectors (f s , f t ), logical prediction outputs (z s , z t ), prediction confidences (p s , p t ) and prediction results (y s , y t )

[0097] (2a) Input the image after data augmentation in Step 1 into the teacher network. Only use the cross-entropy loss L CE to constrain the model during the training process.

[0098] (2b) Input the image after data augmentation in Step 1 into the student network. For each image, obtain the corresponding 64-dimensional feature vector (f s ), further obtain the logical prediction output (z s ) and obtain the student prediction result (y s ). During the training of the student network, use the trained teacher model to calculate the corresponding feature vector (f t ), logical prediction output (zt ) and the predicted results (y t ).

[0099] Step 3: Use three loss terms to optimize the training of the student network model.

[0100] (3a) Using the commonly used cross entropy loss L CE , feature center stable learning loss L SFCL and the selective knowledge distillation loss L SKD The sum of is used to optimize the model training;

[0101] (3b) The feature center stability learning module mainly includes the following steps: i. Using the teacher model to iterate the complete data set to calculate the feature vector (f t ) The average value of the feature vector for each category is obtained to obtain the global category feature center Q g , and use Q g Calculate the affinity matrix A representing the relationship between classes; ii. In the current batch, use the student model to calculate the feature vector of the current batch and calculate the average value of the feature vector for each category to obtain the local (batch) category feature center Q l ; iii. Use the affinity matrix A to first perform feature fusion on the samples in the current batch, thereby improving the feature expression ability of the tail class samples, and then perform feature fusion on the local class feature center Q l Perform weighted optimization (refine) to update the local category feature center; iv. In the current batch, shorten the distance between each sample and its corresponding feature center.

[0102] (3c) The selective knowledge distillation module mainly uses the idea of selective knowledge distillation: the student model selects the label (y s ), the teacher's predicted label (y t ) and the true label (y), focusing on distilling the correct knowledge of the teacher model and filtering out the incorrect knowledge of the teacher model, thereby improving the correctness and reliability of the student model knowledge.

[0103] Experimental results and analysis:

[0104] Experiment 1: Use the present invention to classify images on the CIFAR-10 / CIFAR-100 datasets.

[0105] To verify the effectiveness of the algorithm, an ablation experiment was conducted on the CIFAR-10 / CIFAR-100 test set. Table 1 shows the experimental results. Where 'CE' represents the cross entropy loss L CE , 'SKD' represents the mutual distillation loss L SKD , 'SFCL' represents the inter-distillation loss LSFCL ‘√’ and ‘×’ respectively represent the losses of using and not using the corresponding items. ‘CIFAR-10-Top-1’ and ‘CIFAR-100-Top-1’ respectively represent the average accuracies of the two CIFAR datasets under the condition of an imbalance factor of 100. The experimental results show that the two losses proposed in the present invention have a relatively large degree of performance improvement in the classification tasks of the long-tailed distribution problem, verifying the effectiveness of the method of the present invention.

[0106] Experiment 2: Use the present invention for image classification on the ImageNet2012-LT dataset.

[0107] To verify the effectiveness of the algorithm, it is tested on the ImageNet2012-LT dataset. Table 2 shows the experimental results. It can be found from the results that the long-tailed distribution visual classification method based on sample-aware distillation proposed in the present invention also achieves excellent performance improvement on the ImageNet2012-LT dataset.

[0108] Combining Experiment 1 and Experiment 2, the present invention has significant performance advantages on the existing three long-tailed distribution datasets, surpassing the highest level in the current academic field, verifying that the method proposed in the present invention effectively improves the feature expression ability of tail class samples and successfully selectively distills the effective knowledge of the teacher model.

[0109] Table 1. Ablation experiment of the present invention on the CIFAR-10 / CIFAR-100 test set

[0110]

[0111] Table 2. Test results of the present invention on the ImageNet2012-LT dataset

[0112] ImageNet-Top-1 42.81

[0113] Based on the core idea of mutual distillation, the present invention selectively eliminates the incorrect knowledge of the teacher model while fully learning the teacher distribution by learning to imitate the prediction distribution of the teacher model, making the knowledge learned by the student model more reliable and accurate. Using the idea of mutual distillation, the samples in the same batch pass through two different models, and by selectively fitting each other's feature distributions to make the distribution differences between the two as close as possible, the most correct information is effectively retained.

Claims

1. A visual classification method for long-tailed distributions based on sample-aware distillation, characterized in that It includes the following steps: 1) Establish a long-tailed distribution dataset, sample the dataset as input images and preprocess them; 2) Input the preprocessed images into the teacher network. After training the teacher network model, input all samples of the training set into the pre-trained teacher model to obtain the mean of the feature vectors, calculate the global class feature center, and take out the label with the highest confidence in the prediction confidence to obtain the teacher prediction result; 3) Input the preprocessed images into the student network to train the student network model. The samples pass through the feature encoder to obtain feature vectors. The feature vectors enter two branches, and calculate the cross-entropy loss and the feature center stability learning loss respectively; take out the class label corresponding to the highest confidence in the prediction confidence to obtain the student prediction result; 4) Calculate the selective knowledge distillation loss by using the selective knowledge distillation module with the student prediction result, the teacher prediction result and the true label; The selective knowledge distillation module is used to improve the classification performance of the long-tailed distribution problem. The specific steps are as follows: (4.1) Knowledge distillation uses KL divergence to measure the difference between the distributions obtained after the student model and the teacher model predict the same batch of samples; the specific calculation formula of KL is as follows: where z s and z t are the predicted distributions of the student model and the teacher model, respectively; (4.2) The selective knowledge distillation module selects knowledge based on knowledge distillation; batch_size samples are randomly sampled from each batch and input into two models to obtain the student prediction result y s and the teacher prediction result y t , the student prediction result y s , the teacher prediction result y t and the true label y are used as the inputs of the selective knowledge distillation module to calculate the selective knowledge distillation loss; the distillation weights in this module consist of three cases, and the formula is respectively: ⅰ. When the teacher prediction result is different from the true label and the teacher prediction is wrong, the distillation weight is set to 0; ⅱ. The teacher's prediction result is the same as the student's prediction result, and the distillation weight is reset to p r , where p represents the uncertainty measure of the prediction result, and p = 1 - y t , and r takes 2; ⅲ. When the teacher prediction result is consistent with the true label and the teacher prediction is correct, and the student prediction result is inconsistent with the true label, the distillation weight is set to 1; The total selective knowledge distillation loss is calculated as follows: Among them, τ is the temperature parameter, indicating the weight ratio corresponding to this category calculated according to the number of samples in different categories; By optimizing the selective knowledge distillation loss, selectively reducing the distribution difference between the two models, the student model effectively eliminates the wrong knowledge of the teacher model and improves the classification performance of the student model for the long-tailed distribution problem; The total loss function is as follows: L total = L CE + L SKD + α·L SFCL (9) where α is the balance coefficient of the loss; repeat the set number of iterations until the training ends; 5) Optimize and train the student network model by combining the cross-entropy loss, the feature center stability learning loss and the selective knowledge distillation loss.

2. The long-tailed distribution visual classification method based on sample-aware distillation according to claim 1, wherein In step 1), the long-tailed distribution dataset y i ∈{1,2,...,C}, where N and C represent the total number of image samples and the total number of categories respectively; for this dataset, the number of samples corresponding to different categories is unbalanced. For the c-th category, if the corresponding number of samples is n c c min <...<n c <...<n max ; The number of pictures of the set batch_size is randomly selected as the input of the network for each batch.

3. The long-tailed distribution visual classification method based on sample-aware distillation according to claim 1, characterized in that In step 1), the preprocessing includes normalizing the input images, randomly cropping them to a fixed size, and randomly flipping them for data augmentation.

4. The long-tail distribution visual classification method based on sample-aware distillation according to claim 1, characterized in that In step 2), the specific steps for training the teacher network model are as follows: Input the preprocessed image into the teacher network. The image passes through the feature encoder of the network to obtain a 64-dimensional feature vector f t and the network logic prediction output z t , and this logic prediction output z t is normalized through the functional function to obtain the prediction confidence p t , and this confidence p t and the true label y form the cross-entropy loss L CE to constrain the model; After the training is completed, all samples in the training set are input into the pre-trained teacher model to obtain the feature vectors f′ of all samples t , the logical prediction output z′ t and the prediction confidence p′ t ; using the feature vectors f′ t to calculate the mean of the feature vectors for each category to obtain the global category feature center Q g ; extract the label with the highest confidence in the prediction confidence p′ t to obtain the teacher prediction result y t .

5. The long-tail distribution visual classification method based on sample-aware distillation according to claim 1, wherein In step 3), the specific steps for training the student network model are: Input the image into the student network, where the model of the student network is the same as that of the teacher network, and the student network and the teacher network do not share parameters; the feature encoder of the student network processes each batch of images to obtain a 64-dimensional feature vector f s , and this feature vector f s enters two branches; the first branch further obtains the network logic prediction output Z S , and after passing through the functional function , it is normalized to obtain the prediction confidence p s , and this prediction confidence p s and the true label y are used to calculate the cross-entropy loss L CE ; the second branch obtains the local class feature center Q by calculating the class feature mean of this feature vector l , and together with the global class feature center Q g , they are input into the feature center stability learning module to calculate the feature center stability learning loss L SFCL ; extract the class label corresponding to the highest confidence in the prediction confidence p s to obtain the student prediction result y s ; the student prediction result y s , the teacher prediction result y t , and the true label y are used by the selective knowledge distillation module to calculate the selective knowledge distillation loss L SKD .

6. The long-tail distribution visual classification method based on sample-aware distillation according to claim 5, wherein In step 3), the feature center stability learning module is used to improve the feature expression ability of the tail class samples. The specific steps are as follows: (3.1) For the training samples, the feature vector f of the training set images output by the teacher network t , after logical prediction, gets z t , as shown in Equation (1); z t passes through the softmax function to obtain the predicted confidence p t , as shown in Equation (2); the prediction result y t is the class label corresponding to the maximum value of all predicted confidences, as shown in Equation (3); inputting a batch of image sets into the student network, the corresponding outputs are the feature vector f s , the logical prediction output z s and the prediction result y s , and the calculation process is as follows: z t = logits(f t ), z s = logits(f s ) (1) p t = softmax(z t ), p s = softmax(z s ) (2) Among them, y t = argmax(p t ), y s = argmax(p s )(3) where the argmax function represents taking out the index corresponding to the highest confidence in the prediction confidence, that is, the class label; (3.2) Use the feature center stability learning module to improve the feature expression ability of the tail class samples; it includes the following steps: ⅰ. Use the global class feature center Q obtained by the teacher model g as the input of the feature center stability learning module, and use Q g to calculate the affinity matrix representing the inter-class relationship Furthermore, use the softmax function to normalize the affinity matrix A after removing the diagonal to obtain the normalized affinity matrix ⅱ. Utilize the normalized affinity matrix Perform feature fusion on the feature vectors f of the current batch of samples batch That is Thereby achieving the purpose of enhancing the samples of the tail classes; ⅲ. Utilize the normalized affinity matrix Perform weighted optimization on the local class feature center Q of the current batch l That is Use the method of moving weighted average EMA, and utilize the optimized local class feature center And the global class feature center Q g Update the local class feature center of the current batch in each batch to obtain the optimized local class feature center Q r , and the calculation process formula is as follows: ⅳ. Utilize the generated Q r Reduce the distance between the feature of each sample and its corresponding feature center, making it align with its corresponding feature center. The Manhattan distance is used as the distance metric, that is, the sum of the absolute values of the differences in each dimension of the two vectors, to obtain the loss function of the feature center stable learning module, as shown in the following formula: L SFCL = ||θ(σ(Q r ), σ(f batch ), y)||1 (6) Among them, || ||1 represents the L1 norm, σ(a) = a / ||a|| represents using the Euclidean distance as the distance metric, that is, the L2 norm of the difference between two vectors, θ(b, c) represents the cosine similarity between b and c, and f batch represents the sample features of the current batch.

7. The long-tail distribution visual classification method based on sample-aware distillation according to claim 1, characterized in that In step 5), three loss networks are optimized and trained. During the entire training process of the student network model, the joint cross-entropy loss L CE , the feature center stability learning loss L SFCL , and the selective knowledge distillation loss L SKD are used to optimize and train the network. The cross-entropy loss L CE is used to constrain the model. The feature center stability learning loss L SFCL is used to help enhance the feature expression ability of tail class samples. The selective knowledge distillation loss L SKD is used to help distill knowledge to the student network in a focused manner to improve the classification performance of the long-tail distribution problem. After training, the feature expression ability of the student network model on tail class samples is improved, and it learns the correct knowledge of the teacher network model.

Images