Long tail identification method based on specific category expert knowledge distillation

Abstract

The invention discloses a long tail identification method based on specific category expert knowledge distillation, which comprises the following steps: step 1, interference suppression multi-expert training: dividing data of long tail distribution into three parts, namely a head category, a middle category and a tail category; and step 2, integrating the classification advantages of the expert models in specific categories into a student model (EKDSC) through a knowledge distillation technology. The invention provides a solution with high performance and expandability, the problem of unbalanced head and tail classification performance can be effectively solved, and a new thought is provided for the recognition task of long-tail distributed data. After the EKDSC method is adopted for simple long-tail image classification tasks CIFAR-10LT and CIFAR-100LT, the classification precision is improved by 1-7%, in a natural image ImageNet, the recognition precision of rare species images is improved by 1.2%, the overall recognition precision is improved by 2%, and meanwhile, the reduction of the recognition precision of common species (head classes) is relieved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and long-tail recognition, and can be used in long-tail recognition tasks such as natural species recognition, medical image recognition, and rare scene recognition in autonomous driving. Background Art

[0002] Long-Tailed Recognition (LTR) is an important technology to deal with the problem of data imbalance and is widely used in real-world scenarios, such as the identification of rare species in natural images, the classification or detection of rare lesion images in medical images, etc. In such scenarios, a large amount of data is concentrated in a few "head categories", while the "tail categories" are difficult to train effectively due to the scarcity of samples. Taking natural images as an example, the acquisition of samples of tail categories (such as rare species) is limited by the rarity of species in nature, which further exacerbates the problem of uneven data distribution.

[0003] To solve this problem, early research focused on enhancing the sensitivity of classifiers to the tail image categories. Typical methods include resampling and reweighting techniques. Resampling balances head-class images and tail-class images by adjusting the sample distribution, while reweighting improves the classification performance of tail-class images by modifying the class weights. These methods have improved the recognition accuracy of tail-class images to a certain extent, but usually at the expense of the recognition accuracy of head-class images, and it is difficult to optimize the performance of majority and minority classes at the same time. Subsequently, some ensemble learning methods were proposed. These methods improve the model recognition performance on long-tail image data by integrating multiple complementary classifiers or combining multi-branch architectures with different sampling strategies. However, due to the scarcity of tail category image information, the performance improvement of these methods is still subject to certain limitations. In addition, some work introduces knowledge distillation into long-tail recognition tasks to supplement external image feature information. This type of method distills the knowledge of the teacher model into the student model to enhance the classification ability of the student model, thereby improving the performance of the tail class. However, how to design the teacher model and how to distill the knowledge of the teacher model to the student are still issues that need to be further addressed by this type of method.

[0004] In view of the above problems, the present invention proposes a long-tail recognition method based on category-specific expert knowledge distillation. This method designs and trains expert teacher models that focus on the head class, middle class, and tail class by optimizing the main item suppression interference item mechanism, and fully explores its classification advantages in specific categories. Then, through the classifier-based category-specific knowledge distillation strategy, the knowledge of these expert models is distilled to the classifiers of the corresponding categories respectively to improve the feature extraction and classification capabilities of the student model. This method supplements the problem of scarce knowledge of tail class samples, and at the same time achieves a performance balance for the head, middle, and tail categories, providing an efficient and scalable solution for image classification tasks in long-tail distribution scenarios. Summary of the invention

[0005] The present invention proposes a long-tail recognition method for practical application scenarios, called "Long-Tailed Recognition Based on Expert Knowledge Distillation for Specific Categories" (EKDSC). This method can effectively solve the long-tail classification problems in the fields of recognition of rare species in natural images and medical imaging, improve the performance of the tail category, and take into account the overall performance of the head, middle and tail categories. EKDSC includes a multi-expert model optimized for the head class, middle class and tail class, and a multi-classifier knowledge distillation framework. The multi-expert model based on main item optimization and interference suppression enables each category expert to focus on its specific field while reducing interference from other categories, thereby enhancing the recognition ability of the teacher model. At the same time, the effective knowledge transfer from experts in a specific category to students in the corresponding category significantly improves the long-tail recognition performance of the student model.

[0006] To achieve the above object, the present invention adopts the following technical solution:

[0007] Step 1: Interference suppression multi-expert training divides the data of the long-tail distribution into three parts: head class, middle class and tail class. For each expert model, define its main optimization item (focus on the category) and interference suppression item (influence of other categories), and impose constraints on different categories through a dynamic weight allocation mechanism. For example, for the problem of identifying rare lesions in medical images, the main optimization item of the tail expert is the feature learning of rare lesion categories, while the interference suppression item reduces the impact of common lesions on feature learning through regularization.

[0008] Step 2: Through knowledge distillation technology, the classification advantages of each expert model on specific categories are integrated into a student model (EKDSC). In this process, on the one hand, the knowledge of the specific expert model is distilled to the corresponding classifier of the student model, and on the other hand, a delayed reweighting strategy is applied to enable the student model to better optimize the performance of minority categories based on learning expert knowledge, thereby improving the global classification accuracy.

[0009] Traditional long-tail classification methods usually need to make a trade-off between optimizing the performance of the head class and the tail class. The improvement of the tail class performance is often at the expense of the accuracy of the head class. The present invention improves the classification capabilities of the head, middle, and tail categories respectively through interference suppression multi-expert model training; and through the student model distillation based on expert knowledge, the specific advantages of the multi-expert model are efficiently integrated into the student model, thereby significantly improving the overall performance of the model on long-tail distribution data. The present invention provides a solution with both high performance and scalability, which can effectively solve the problem of imbalanced classification performance between the head and tail classes, and provides a new idea for the recognition task of long-tail distribution data. In the simple long-tail image classification tasks CIFAR-10LT and CIFAR-100LT, after using the EKDSC method, a 1-7% improvement in classification accuracy was achieved. In the natural image ImageNet, the recognition accuracy of rare species images was improved by 1.2%, and the overall recognition accuracy was improved by 2%, while the decline in recognition accuracy of common species (head class) was alleviated. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 The framework diagram of multi-expert network for interference suppression;

[0011] Figure 2 This is the framework diagram of the expert knowledge distillation network;

[0012] Figure 3 Comparison of head, middle, and tail class expert performance before and after using prime term optimization and distractor suppression. DETAILED DESCRIPTION

[0013] Introduction to the network framework: It is divided into interference suppression multi-expert network and expert knowledge distillation network. Interference suppression multi-expert (ISME) network: It aims to obtain an expert model of a specific category by optimizing the main items and suppressing the interference items. In the present invention, different experts correspond to different main items and interference items. The present invention introduces mask technology to adjust the output score to achieve the goal of optimizing the main items and suppressing the interference items. Taking the tail class as an example, the main item of the tail class expert is the category contained in the tail, and the categories corresponding to the head and the middle are interference items, such as Figure 1 As shown, for the tail class expert model, the category score of the tail class main item remains unchanged, while the scores of other interference items are set to zero; at the same time, the output scores of the head and middle item categories should be as small as possible. In order to express clearly, the present invention first makes some symbol definitions.

[0014] Given a long-tail image dataset Here we take the natural image ImageNet as an example, where X represents the image data, Y represents the category label corresponding to the current image, and there are a total of C categories, forming a class set Divide the C ImageNet image categories into three subsets: head, middle and tail (common species image categories), (more common species image categories), (rare species image category), where The data subsets corresponding to the class subsets are (common species image collection), (Rare species image collection), the corresponding experts are head, middle and tail class experts ε H (Expert in processing images of common species),ε M ,ε T (Expert in processing images of rare species).

[0015] Set the data subsets of the main term and interference term to and The corresponding class sets are and Here we take the tail class expert ε T For example, its main item class set is The set of interference items is in is interference term 1, is interference term 2.

[0016]

[0017]

[0018]

[0019] Accordingly, the expert ε * (ε H ,ε M ,ε T The output score (logits) of one of them is expressed as: and During the training process of the expert model for a specific category, the loss comes from three experts. The loss function of each expert consists of the main item optimization loss and the interference item suppression loss. Since the main item optimization data of each expert model corresponds to the corresponding category Therefore, the optimization data of the main items of different expert models are unbalanced, so the optimization of the main items uses the score-adjusted balanced binary cross entropy (Bal_BCE) loss function. * (ε H ,ε M ,ε T One of them), then the loss function of the dominant term optimization is:

[0020]

[0021] Here the weight value is set to w r =1, represents the indicator function, by Figure 1 For example, Mask_Tail_Expert is 1 1×C The matrix after setting the interference item categories to zero, C represents the number of categories in the current data set. Represents class y in the main item r distribution, so there is a bias term for the test prior for,

[0022]

[0023] For experts, interference items are usually the main source of confusion, so the misjudgment of interference items should be minimized in the process of classifying and judging the dominant items. In order to eliminate the confusion of experts caused by interference items and enable each expert to work together in a complementary way, we have A loss function is designed to suppress the score output of interference items:

[0024]

[0025]

[0026] Where N I1 and N I2 is the total sample size of each interference item, is 1 1×C After setting the main item categories to zero, the 2 in the lower right corner represents the 2-norm.

[0027] The interference suppression loss plays a role of soft regularization in the expert optimization process, minimizing the scores of class samples in non-optimized main items, thereby reducing their influence on the classification of optimized main items. In general, for a single expert ε * (ε H ,ε M ,ε T One of them), its loss function is:

[0028]

[0029] The total expert loss function is:

[0030]

[0031] Expert knowledge distillation network: Figure 2As shown in the figure, the teacher model of knowledge distillation is the head, middle and tail experts trained in the interference suppression multi-expert network. The student model uses CeiT as the backbone network and designs three classifiers, namely the head, middle and tail classifiers. Each expert in a specific category is connected to the corresponding classifier of the student model to guide the training of the student model. Therefore, the loss function of the student model includes two parts, namely the classification loss of the model itself and the distillation loss of the teacher model on the student model.

[0032] The knowledge distillation loss constrains the output score of the student network to be as close to the teacher network as possible after softmax processing. The loss of the student model itself first uses the cross entropy loss constraint to obtain a good generalization feature that focuses relatively on the majority class; then the Bal_BCE loss is used to improve the performance of the minority class by applying a larger weight to the minority class. In order to calculate the loss of the student model itself, since each classifier of the student model will output a score, we use the LogitAverager component to aggregate and average the output scores of the three classifiers in the category dimension. The final score is output as:

[0033]

[0034] in There are three classifiers for the head, middle and tail classes for sample x i The output scores of . Further, the classification loss of the student model is calculated based on these scores.

[0035] The final overall loss function is:

[0036]

[0037] in Represents the classification loss of the student model, that is, first use loss, and then use the formula (10) after a certain number of training rounds represents the knowledge distillation loss, in, and are the distillation losses of the head, middle and tail class experts respectively. The specific information is shown in formula (11). λ is a hyperparameter to balance different losses.

[0038] Next, we and distillation losses Give a detailed description:

[0039]

[0040] The weight value is set to w i =1, Represents the indicator function.

[0041]

[0042] in express and One of them, i * Represents sample x i The scores after being processed by the teacher (one of the head, middle, and tail class experts) model, converted to one-hot encoding. It represents the predicted category probability of one of the head, middle and tail class classifiers of the student model for the current sample, that is, the probability after the classifier output score is processed by softmax.

[0043] Experimental Section

[0044] Experimental datasets: This paper is evaluated on two small image recognition datasets, CIFAR-10LT and CIFAR-100LT, and ImageNet-LT, a large natural image recognition dataset.

[0045] CIFAR-10LT and CIFAR-100LT are subsets of CIFAR-10 and CIFAR-100, respectively. Both CIFAR-10 and CIFAR-100 contain 50,000 images for training and 10,000 images for validation, with image size of 32×32 and containing 10 and 100 categories, respectively. The data imbalance factor ρ is defined as (The ratio of the number of samples of the category with the most samples to the number of samples of the category with the least samples). This value reflects the degree of imbalance in the data. The imbalance factors in the experiment are set to 100 and 50.

[0046] ImageNet-LT is a long-tail version of the original ImageNet, a subset obtained by sampling from a Pareto distribution with a power value of α = 0.6. The dataset contains a total of 115.8K images, covering 1000 categories, with the number of images in each category ranging from 1280 to 5.

[0047] Evaluation Metrics:

[0048] We use Top-1 Accuracy as the evaluation indicator.

[0049] Experimental setup: This paper is built on the PyTorch framework and trained on two RTX 4090 GPUs, with the batch size set to 128. For the small-scale datasets CIFAR-10LT and CIFAR-100LT, a total of 700 rounds of training were performed, of which the first 500 rounds used cross entropy loss (CE) and the last 200 rounds used balanced binary cross entropy loss (Bal_BCE). For ImageNet-LT, a total of 1400 rounds of training were performed, of which the first 800 rounds used cross entropy loss (CE) and the last 600 rounds used balanced binary cross entropy loss (Bal_BCE). During data preprocessing, the training samples were first resized to 256×256 and then center cropped to 224×224. Similarly, the test samples were also center cropped to 224×224. In terms of optimizer, the AdamW optimizer was used, and the cosine learning rate scheduler was used, and the initial learning rate was set to 5×10 -4 , weight decay is set to 1×10 -5 .

[0050] Comparative experiments: The present invention is evaluated against some baseline methods and the most advanced long-tail recognition methods on CIFAR-10LT, CIFAR-100LT and ImageNet-LT datasets, all of which are trained from scratch. Table 1 shows the quantitative results on CIFAR-10LT and CIFAR-100LT datasets, and Table 2 shows the quantitative results on ImageNet-LT dataset, all of which achieve the best results.

[0051] Table 1. Results on the CIFAR-10LT and CIFAR-100LT datasets (ρ=50 and ρ=100). This table compares the performance of different methods in terms of overall classification Top-1 accuracy (%). The best results are marked in bold.

[0052]

[0053] Table 2. Results of the ImageNet-LT dataset. This table shows the overall classification Top-1 accuracy (%) of different methods on this dataset. The best results are marked in bold.

[0054]

[0055] Ablation experiment: The present invention conducts a series of ablation experiments on the datasets CIFAR-10LT and CIFAR-100LT (imbalance factor ρ=100) to verify the role of each module.

[0056] 1) Based on the performance analysis of CeiT of different scales, Table 3 shows the results of EKDSC experiments using the Tiny, Small and Base architectures of the CeiT model.

[0057] Table 3. Analysis of the performance impact of CeiT models of different sizes on the EKDSC method on the CIFAR-10LT and CIFAR-100LT (ρ=100) datasets.

[0058]

[0059] 2) Performance analysis of different components in the EKDSC model. First, only the baseline network CeiT is used, then EKDSC without knowledge distillation is used, and then the method of expert knowledge distillation for specific categories is added on the basis of EKDSC. The experimental results are shown in Tables 4 and 5.

[0060] Table 4. Ablation study of different components (CIFAR-100LT, ρ=100), the best results are marked in bold.

[0061]

[0062] Table 5. Ablation study of different components (CIFAR-10LT, ρ=100), the best results are marked in bold.

[0063]

[0064] 3) ISME performance analysis,First, the performance differences before and after applying main item optimization and interference item suppression to the head, middle and tail experts are analyzed, such as Figure 3 Next, the performance difference between the EKDSC of the distilled expert model and the method of directly applying the main term optimization and interference term suppression constraints to the student model is analyzed. This method is named ISME-LT, and the experimental results are shown in Table 6.

[0065] Table 6. Performance comparison of ISME-LT and EKDSC (CIFAR-100LT, ρ = 100), the best results are marked in bold.

[0066]

[0067] 4) Hyperparameter analysis in the loss function of the EKDSC model. As shown in Table 7, the model performance is best when the hyperparameter λ is close to 0.5.

[0068] Table 7. The influence curve of the hyperparameter λ on the model performance in (EKDSC model loss function) (CIFAR-100LT, ρ=100), and the optimal result is marked in bold.

[0069]

Claims

1. A long-tail identification method based on distillation of expert knowledge of a specific category, characterized in that: The steps include: Step 1: Interference suppression multi-expert training divides the long-tail distribution data into three parts: head class, middle class and tail class; for each expert model, define its main optimization items and interference suppression items, and impose constraints on different categories through a dynamic weight allocation mechanism; Step 2: Through the knowledge distillation technology, the classification advantages of each expert model on a specific category are integrated into a student model EKDSC; the knowledge of the specific expert model is distilled to the corresponding classifier of the student model; by applying the delayed reweighting strategy, the student model can better optimize the performance of the minority category based on the expert knowledge, thereby improving the global classification accuracy; Through multi-expert model training with interference suppression, the classification capabilities of the head, middle and tail categories are improved respectively; and through student model distillation based on expert knowledge, the specific advantages of the multi-expert model are efficiently integrated into the student model, thereby significantly improving the overall performance of the model on long-tail distribution data.

2. According to claim 1, a long-tail identification method based on category-specific expert knowledge distillation is characterized in that: The network framework of long-tail recognition is divided into interference suppression multi-expert network and expert knowledge distillation network; Interference suppression multi-expert ISME network: By optimizing the main items and suppressing the interference items, an expert model of a specific category is obtained; different experts have different main items and interference items, and the mask technology is introduced to adjust the output score to achieve the goal of optimizing the main items and suppressing the interference items; taking the tail class as an example, the main item of the tail class expert is the category contained in the tail, while the categories corresponding to the head and the middle are interference items. For the tail class expert model, the category score of the tail class main item remains unchanged, while the scores of other interference items are set to zero; at the same time, the output scores of the head and middle item categories should be as small as possible; Given a long-tail image dataset Here we take the natural image ImageNet as an example, where X represents the image data, Y represents the category label corresponding to the current image, and there are a total of C categories, forming a class set Divide the C ImageNet image categories into three subsets: head, middle and tail That is, common species image categories, That is, the more common species image categories, That is, rare species image category, where The data subsets corresponding to the class subsets are That is, a collection of common species images, That is, the rare species image set, the corresponding experts are the head, middle and tail class experts ε H That is, an expert in processing images of common species, ε M ,ε T That is, experts in processing images of rare species; Set the data subsets of the main term and interference term to and The corresponding class sets are and Here we take the tail class expert ε T For example, its main item class set is The set of interference items is in is interference term 1, is interference item 2; Accordingly, the expert ε * (ε H ,ε M ,ε T The output score of one of them is expressed as: and During the training of the category-specific expert model, the losses come from three experts; the loss function of each expert consists of the main item optimization loss and the interference item suppression loss; The optimization of the main term uses the score-adjusted balanced binary cross entropy loss function; for the expert ε * (ε H ,ε M ,ε T One of them), then the loss function of the dominant term optimization is: Here the weight value is set to w r =1, represents the indicator function, 1 1×C The matrix after setting the interference item categories to zero, C represents the number of categories in the current data set; set Represents class y in the main item r distribution of , with a bias term for the test prior for, In order to eliminate the confusion of interference items on the experts' classification and enable each expert to work together in a complementary way, A loss function is designed to suppress the score output of interference items: Where N I1 and N I2 is the total sample size of each interference item, is 1 1×C After setting the main item category to zero, the 2 in the lower right corner represents the 2-norm; The interference suppression loss plays a role of soft regularization in the expert optimization process, minimizing the scores of category samples in the non-optimized main items, thereby reducing the impact on the classification of the optimized main items; in general, for a single expert ε * (ε H ,ε m ,ε T One of them), its loss function is: The total expert loss function is: Expert knowledge distillation network: The teacher model of knowledge distillation is the head, middle and tail category experts trained in the interference suppression multi-expert network. The student model uses CeiT as the backbone network and designs three classifiers, namely the head, middle and tail classifiers. Each specific category of experts is connected to the corresponding classifier of the student model to guide the training of the student model. Therefore, the loss function of the student model includes two parts, namely the classification loss of its own model and the distillation loss of the teacher model on the student model.

3. The long-tail identification method based on category-specific expert knowledge distillation according to claim 2, characterized in that: The loss of the student model itself is first constrained using the cross entropy loss to obtain a good generalization feature that focuses on the majority class; Then, the Bal_BCE loss is used to improve the performance of the minority class by applying a larger weight to the minority class. In order to calculate the loss of the student model itself, since each classifier of the student model outputs a score, the output scores of the three classifiers in the category dimension are summarized and averaged through the LogitAverager component. The final score is output as: in There are three classifiers for the head, middle and tail classes for sample x i The output score of Calculate the classification loss of the student model based on the scores; The final overall loss function is: in Represents the classification loss of the student model, that is, first use loss, and then use the formula (10) after a certain number of training rounds represents the knowledge distillation loss, in, and are the distillation losses of the head, middle and tail class experts respectively. The specific information is shown in formula (11). λ is a hyperparameter to balance different losses. Next, and distillation losses Give a detailed description: The weight value is set to w i =1, represents the indicator function; in express and One of them, i * Represents sample x i The scores after being processed by the teacher model are converted into one-hot encodings; It represents the predicted category probability of one of the head, middle and tail class classifiers of the student model for the current sample, that is, the probability after the classifier output score is processed by softmax.

Images