Long-tail image classification method and system based on multi-sample feature migration
Through multi-sample feature transfer and mutual differentiation processing, the feature information of the header sample is transferred to the tail sample, which solves the problem of poor classification of tail sample on the long-tail data set by the deep learning model, and improves the classification accuracy and Class spacing achieves better long-tail image classification effect.
Patent Information
- Application Number
- CN202411312224.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2025-01-03
AI Technical Summary
Deep learning models perform poorly on long-tail datasets, especially in poor classification of tail samples. Existing methods such as class balance, decoupling learning and multi-branch models have risks of overfitting or complex hyperparameter adjustment, which is difficult to be effective. Improve the classification accuracy of the tail category.
Through multi-sample feature migration and mutual differentiation processing, the feature information of the header sample is fused into the tail-class samples, and the full connection layer classifier is used to train, so as to improve the model's attention to the tail-class samples, and to maintain attention to the header sample, and improve the Classification accuracy and inter-class spacing.
It effectively improves the classification accuracy of tail samples, reduces intra-class spacing, and maintains attention to the characteristics of head samples, improving the overall classification effect of long-tail data sets.
Smart Images
Figure CN119251567A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image classification technology, and in particular to a long-tail image classification method and system based on multi-sample feature migration. Background Art
[0002] Deep learning is widely used in various visual tasks. Generally speaking, deep learning models can only perform well on large and evenly distributed datasets. However, real-world data usually presents a long-tail distribution, which limits the performance and application of deep learning on long-tail datasets. In the dataset of long-tail image classification, the number of samples in the head class is the largest, and the number of samples in the tail class is the smallest. Such data distribution will cause model deviation during training, affecting the classification effect of the tail class.
[0003] At present, researchers have proposed many methods to solve the sample imbalance problem of long-tail datasets, including: class balancing methods, decoupled learning methods, and multi-branch model methods. Class balancing methods include resampling and reweighting. Although resampling and reweighting methods can improve the overall classification accuracy, there is a risk of model overfitting or information loss in certain categories; decoupled learning methods can improve the classification performance of tail categories, but this two-stage strategy requires complicated hyperparameter adjustments in practice; the existence of multi-branch models helps to explore the complementarity between branches, but it does not necessarily significantly improve the recognition performance of a single branch model. Therefore, in long-tail image classification, it is necessary to further explore new methods to solve the classification problem of tail categories, improve the classification accuracy of tail categories while taking into account the overall accuracy. Summary of the invention
[0004] In order to solve the problems existing in the prior art, the present invention proposes a long-tail image classification method and system based on multi-sample feature migration, which can effectively improve the classification effect of long-tail data sets by integrating the rich feature information of head class samples into tail class samples, thereby increasing the classification model's attention to tail class samples.
[0005] The technical solution adopted by the present invention is: a long-tail image classification method based on multi-sample feature migration, comprising the following steps:
[0006] S1, extracting original features from the training samples of the long-tail dataset input in batches;
[0007] S2, performing multi-sample feature migration processing on the original features;
[0008] S3, performing multi-sample feature difference processing on the original features;
[0009] S4, according to the output features obtained by multi-sample feature migration processing and the different features obtained by multi-sample feature difference processing, the sample features are trained and classified using a fully connected layer classifier to obtain a training prediction output result;
[0010] The long-tail image classification method migrates the feature information of the head class samples of the long-tail image to the tail class samples by performing multi-sample feature migration and mutual difference processing on the original features, so that the focus of the classification model used to classify the long-tail image is shifted while maintaining the focus on the original head class sample feature information.
[0011] The following technical solution is also adopted in the embodiment of the present invention: a long-tail image classification system based on multi-sample feature migration, comprising the following modules:
[0012] The backbone network is used to extract original features from the training samples of the long-tailed dataset input in batches;
[0013] Multi-sample feature migration module, which performs multi-sample feature migration processing on the original features;
[0014] Multi-sample feature difference module, which performs multi-sample feature difference processing on the original features;
[0015] A training module is used to train and classify sample features using a fully connected layer classifier based on output features obtained by multi-sample feature migration processing and different features obtained by multi-sample feature different processing, so as to obtain training prediction output results;
[0016] The long-tail image classification system migrates the feature information of the head class samples of the long-tail image to the tail class samples by performing multi-sample feature migration and mutual difference processing on the original features, so that the focus of the classification model used to classify the long-tail image is shifted while maintaining the focus on the original head class sample feature information.
[0017] Compared with the prior art, the technical effects achieved by the present invention include:
[0018] The present invention operates on the original features of the samples and integrates the rich feature information of the head class samples into the tail class samples, thereby increasing the model's attention to the tail class samples and improving the classification accuracy of the tail class. In addition, the present invention can increase the inter-class spacing of the head class samples and the tail class samples and reduce the intra-class spacing. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 Schematic diagram of the main process of the long-tail image classification method for multi-sample feature migration in an embodiment of the present invention.
[0020] Figure 2 Schematic diagram of a multi-sample feature migration framework in an embodiment of the present invention.
[0021] Figure 3 Schematic diagram of a framework with different features for multiple samples in an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to enable ordinary technicians in the field to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the specification; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the claims of the present invention.
[0023] Example
[0024] This embodiment proposes a long-tail image classification method based on multi-sample feature migration, such as Figure 1 As shown, the following steps are included:
[0025] S1: The training samples of the long-tail dataset are passed into the backbone network in batches. The backbone network extracts rich original features F from the input training samples. of .
[0026] Assuming that the number of training samples passed in each batch is n, the training samples of the long-tail dataset are passed in batches to the backbone network for processing. The backbone network uses the ResNet-32 residual network, which has 32 layers; the core component of each residual network layer is the residual block, each residual block contains two or three convolutional layers, and has a skip connection to bypass these convolutional layers.
[0027] Use symbols Represents the original features obtained for each training sample, then the original features obtained for n training samples in each batch are recorded as The original features reflect the information contained in the training samples and provide a data basis for the operations in subsequent steps.
[0028] S2. Perform multi-sample feature migration processing on the original features.
[0029] In long-tail datasets, the number of head-class samples is large and dominant, so the head-class samples have the largest number of categories in each batch.
[0030] The process of multi-sample feature migration includes the following steps:
[0031] S21. Concatenate the original features obtained from each batch with the remaining original features in the batch to obtain a concatenated feature vector.
[0032] If each batch has n training samples, and a total of n original features are obtained through backbone network extraction, then each of the n original features is spliced with the other n-1 original features in the batch (i.e., the spliced original features) to obtain the spliced feature vector of the batch.
[0033] When two original feature samples are spliced, the size of the original feature samples will not be changed. The two original feature samples are directly spliced according to the number of channels. The number of channels of the feature vector after splicing is twice that of the original feature.
[0034] S22, performing association calculation on the concatenated feature vector through a multi-layer perceptron; the multi-layer perceptron captures the deep relationship in the concatenated feature vector through a series of weight adjustments and nonlinear transformations, and then obtains the correlation score through the first activation function and normalization function.
[0035] In this step, the sigmoid activation function and normalization function are used to obtain n-1 correlation scores, denoted as A i ={a ij |j∈[1,n],j≠i}, where a ij It represents the correlation score obtained after the concatenated feature vector of the i-th original feature and the j-th original feature is processed by the multi-layer perceptron MLP.
[0036] The correlation score reflects the proportion of shared features between two original features. Since in each batch, each original feature is concatenated with the other n-1 original features in the batch, n-1 correlation scores will be obtained after calculation by the multi-layer perceptron. These correlation scores represent the relationship between this original feature and other original features in the batch.
[0037] S23. Multiply the correlation score by its corresponding original feature and add them up to obtain the shared feature.
[0038] In this step, the correlation score a ij The corresponding original features (i.e. the original features concatenated in step S21) are multiplied and accumulated to obtain a new feature vector; the new feature vector represents the shared features extracted from a certain sample from other samples.
[0039] Shared Features The expression is:
[0040]
[0041] Shared features represent the similarities between the features of two samples; Represents the original features to be concatenated. The correlation score is proportional to the similarity between the two sample features.
[0042] S24. A nonlinear relationship is established between the shared features and the original features obtained from each batch through a second activation function, and then the shared features are linearly added to obtain the output features after migration processing.
[0043] In this step, the shared features and the original features are connected through the tanh activation function to establish a nonlinear relationship, which improves the model's ability to express nonlinear features. The shared features are then linearly added to obtain the final output features.
[0044]
[0045] Among them, ε is a preset adjustable hyperparameter. represents the original feature that currently needs to be concatenated with other n-1 original features (i.e., the concatenated original features), and o(·) represents the second activation function.
[0046] S3. Perform multi-sample feature difference processing on the original features.
[0047] During feature migration, shared features are migrated to the original features. This process easily migrates the features of the tail class samples to the features of the head class samples, thereby affecting the model's discrimination of the head class samples and reducing the model's attention to the head class samples. The purpose of mutual difference processing is to try to maintain attention to the original head class sample features.
[0048] For different samples, the original features of the samples can be regarded as the sum of shared features and different features. Different features represent the unique parts of the original features, and can be used to keep the focus on the head class samples. Based on this idea, this embodiment first calculates the different scores {1-a} between each sample based on the correlation scores obtained by multi-sample feature migration. ij}; Based on these different scores, generate different features representing the differences between samples:
[0049]
[0050] Where j∈{1,2,…,n}. This not only considers the correlation score with some other sample, but also extracts different features from the perspective of multiple samples, which is conducive to considering the original features that need to be spliced at present. In this way, the model can maintain effective attention on the features of the head class samples during feature transfer.
[0051] The size of the output features and the mutually different features is the same as the size of the original features, that is, after the multi-sample feature migration process and the multi-sample feature mutual difference process, the size of the sample features finally outputted remains unchanged.
[0052] S4. Based on the output features obtained by multi-sample feature migration processing and the different features obtained by multi-sample feature difference processing, a fully connected layer classifier is used to train and classify the sample features to obtain a training prediction output result.
[0053] In this step, two fully connected layer classifiers are used to train and classify sample features based on the output features obtained by multi-sample feature migration and the different features obtained by multi-sample feature different processing. The first training classifier uses the output features after multi-sample feature migration for training and classification to obtain the first training prediction result, thereby increasing the classification model's attention to the tail class samples. The second training classifier trains and classifies the different features of the samples to obtain the second training prediction result, thereby maintaining the classification model's attention to the head class samples.
[0054] The first training classifier obtains the first training prediction result ρ1 as:
[0055]
[0056] The second training prediction result ρ2 obtained by the second training classifier is:
[0057]
[0058] Add the first training prediction result to the second training prediction result to get the final training prediction output result:
[0059]
[0060] in represents the first trained classifier, represents the second trained classifier, and is a fully connected layer classifier of the same dimension; λ is an adjustable hyperparameter.
[0061] S5. According to the training prediction output results, the loss function is used to optimize the training process of the classification model.
[0062] When the classification model is trained, the training prediction output results obtained in each round are optimized through the loss function. The model training is guided by calculating the loss value between the training prediction output results and the true value, making the model effect better and better.
[0063] Taking the cross entropy loss function as an example, the loss value is calculated as follows:
[0064]
[0065] Where i∈{1,2,=,n}, n is the total number of samples, y i is the true label of the sample.
[0066] The final training classification prediction result obtained during classification model training passes through the loss function. By calculating the loss value between the training classification prediction result and the true value of the sample, we can know the performance of the model under the current parameters. The loss function provides an optimization goal. The classification model adjusts the parameters by minimizing the loss to make the output result more accurate. Among them, the classification model is used to classify images. The specific form of the classification model is not limited and can be implemented using the classification model in the prior art.
[0067] This embodiment also provides a long-tail image classification system based on multi-sample feature migration, including the following modules:
[0068] The backbone network is used to extract original features from the training samples of the long-tailed dataset input in batches;
[0069] Multi-sample feature migration module, which performs multi-sample feature migration processing on the original features;
[0070] Multi-sample feature difference module, which performs multi-sample feature difference processing on the original features;
[0071] A training module is used to train and classify sample features using a fully connected layer classifier based on output features obtained by multi-sample feature migration processing and different features obtained by multi-sample feature different processing, so as to obtain training prediction output results;
[0072] The long-tail image classification system migrates the feature information of the head class samples of the long-tail image to the tail class samples by performing multi-sample feature migration and mutual difference processing on the original features, so that the focus of the classification model used to classify the long-tail image is shifted while maintaining the focus on the original head class sample feature information.
[0073] In the long-tail image classification system of this embodiment, the multi-sample feature migration and differentiation module includes a multi-sample feature migration module and a multi-sample feature differentiation module. Figure 2 , Figure 3 As shown, the original features are subjected to multi-sample feature migration and mutual difference processing, and the information of the head class samples is migrated to the tail class samples, changing the focus of the model. Among them, the multi-sample feature migration module obtains the original features from the backbone network and performs migration processing on them. The multi-sample feature mutual difference module is an improvement on the multi-sample feature migration module. In order to keep the focus on the original head class sample features as much as possible, the correlation score in the multi-sample feature migration module is used to obtain the mutual difference scores between each sample, and then obtain the mutual difference features. The mutual difference features are sent to the second training classifier for training prediction to obtain the second training prediction results.
[0074] The present invention cleverly utilizes all feature information of samples in each batch, and migrates the feature information of the head class samples to the tail class samples through multi-sample feature migration and mutual differentiation, so that the model's focus is shifted, and the focus on the original head class sample feature information is maintained as much as possible, ultimately improving the classification performance and generalization ability of the model. The method of the present invention is only used in the training phase, and the multi-sample feature migration mutual differentiation module is not used in the test phase. Subsequently, the trained model can be used to test and actually classify the long-tail images.
[0075] The above is only the better / preferred implementation method of the invention of the present invention, but the protection scope of the invention patent is not limited to this. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principle of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A long-tail image classification method based on multi-sample feature migration, characterized in that: The following steps are involved: S1, extracting original features from the training samples of the long-tail dataset input in batches; S2, performing multi-sample feature migration processing on the original features; S3, performing multi-sample feature difference processing on the original features; S4, according to the output features obtained by multi-sample feature migration processing and the different features obtained by multi-sample feature difference processing, the sample features are trained and classified using a fully connected layer classifier to obtain a training prediction output result; The long-tail image classification method migrates the feature information of the head class samples of the long-tail image to the tail class samples by performing multi-sample feature migration and mutual difference processing on the original features, so that the focus of the classification model used to classify the long-tail image is shifted while maintaining the focus on the original head class sample feature information.
2. The long-tail image classification method according to claim 1, characterized in that: The process of multi-sample feature migration processing in step S2 includes the following steps: S21, splicing the original features obtained from each batch with the remaining original features in the batch to obtain a spliced feature vector of the batch, wherein the remaining original features are the spliced original features; S22, performing association calculation on the concatenated feature vector through a multi-layer perceptron; the multi-layer perceptron captures the deep relationship in the concatenated feature vector through a series of weight adjustments and nonlinear transformations, and then obtains the correlation score through a first activation function and a normalization function; S23, multiplying the correlation score by the concatenated original features and accumulating the results to obtain a shared feature; S24. A nonlinear relationship is established between the shared features and the original features obtained from each batch through a second activation function, and then the shared features are linearly added to obtain the output features after migration processing.
3. The long-tail image classification method according to claim 2, characterized in that: Shared Features The expression is: Among them, n is the number of training samples passed in each batch, a ij Represents the correlation score obtained after the concatenated feature vector of the i-th original feature and the j-th original feature is processed by the multi-layer perceptron; Represents the original features being concatenated.
4. The long-tail image classification method according to claim 2, characterized in that: Output Features The expression is: Among them, ε is a preset adjustable hyperparameter. For shared features, represents the original feature that needs to be concatenated with other n-1 original features, n is the number of training samples passed in for each batch, and o(·) represents the second activation function.
5. The long-tail image classification method according to claim 1, characterized in that: In the process of multi-sample feature difference processing in step S3, the difference scores between each sample are first calculated according to the correlation scores obtained by multi-sample feature migration; and then, the difference features representing the differences between the samples are generated according to the difference scores.
6. The long-tail image classification method according to claim 5, characterized in that: Different features The expression is: Among them, j∈{1,2,…,n}, n is the number of training samples passed in each batch, a ij It represents the correlation score obtained by processing the concatenated feature vector of the i-th original feature and the j-th original feature by the multi-layer perceptron. Indicates the original feature that currently needs to be spliced with other n-1 original features.
7. The long-tail image classification method according to claim 1, characterized in that: In step S4, the fully connected layer classifier includes a first training classifier and a second training classifier; The first training classifier uses the output features after the multi-sample feature migration to perform training classification, and obtains the first training prediction result, so as to improve the classification model's attention to the tail class samples; The second training classifier performs training classification on the mutually different features to obtain the second training prediction result, thereby keeping the classification model focused on the first class samples; Add the first training prediction result to the second training prediction result to get the final training prediction output result: in represents the first trained classifier, represents the second training classifier, λ is an adjustable hyperparameter, is the output feature, For different characteristics.
8. The long-tail image classification method according to claim 1, characterized in that: The long-tail image classification method further comprises the steps of: S5. According to the training prediction output results, the loss function is used to optimize the training process of the classification model.
9. A long-tail image classification system based on multi-sample feature migration, characterized in that: Includes the following modules: The backbone network is used to extract original features from the training samples of the long-tailed dataset input in batches; Multi-sample feature migration module, which performs multi-sample feature migration processing on the original features; Multi-sample feature difference module, which performs multi-sample feature difference processing on the original features; A training module is used to train and classify sample features using a fully connected layer classifier based on output features obtained by multi-sample feature migration processing and different features obtained by multi-sample feature different processing, so as to obtain training prediction output results; The long-tail image classification system migrates the feature information of the head class samples of the long-tail image to the tail class samples by performing multi-sample feature migration and mutual difference processing on the original features, so that the focus of the classification model used to classify the long-tail image is shifted while maintaining the focus on the original head class sample feature information.
10. The long-tail image classification system according to claim 9, characterized in that: The migration process of the multi-sample feature migration module includes: The original features obtained from each batch are concatenated with the remaining original features in the batch to obtain a concatenated feature vector of the batch, wherein the remaining original features are the concatenated original features; The concatenated feature vector is associated with a multi-layer perceptron. The multi-layer perceptron captures the deep relationship in the concatenated feature vector through a series of weight adjustments and nonlinear transformations, and then obtains the correlation score through the first activation function and normalization function. Multiply the correlation score with the concatenated original features and add them together to obtain the shared features; A nonlinear relationship is established between the shared features and the original features obtained in each batch through the second activation function, and then the shared features are linearly added to obtain the output features after migration processing.