Long-tail visual recognition methods, devices, and electronic equipment based on feature aggregation loss.
By using the feature aggregation loss method, the feature distribution of the long-tailed visual recognition model is adjusted, which solves the problems of model overfitting and unclear features, and achieves higher recognition accuracy and feature extraction capability.
Patent Information
- Application Number
- CN202311372977.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-20
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-10-20
AI Technical Summary
Existing long-tail visual recognition models are prone to overfitting to tail categories during training, resulting in low recognition accuracy and unclear feature extraction, making them difficult to apply effectively in the real world.
By using the feature aggregation loss method, the distribution of features in space is adjusted by utilizing the feature balancing factor and hard sample feature constraints, so that the distance between features of different categories is more balanced and similar features are clustered together, thereby improving the recognition accuracy of the model.
It enhances the model's ability to identify tail-class samples, improves feature extraction capabilities and recognition accuracy, and makes the class distribution boundaries of features in space clearer, thereby improving the overall recognition performance of the model.
Smart Images

Figure CN117611857B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of visual recognition, specifically relating to a long-tail visual recognition method, device, and electronic device based on feature aggregation loss. Background Technology
[0002] In the real world, data often exhibits a long-tail distribution, meaning a few classes have a large number of samples, while most classes have only a small number. This imbalance can cause models trained on such datasets to overfit to the smaller tail classes. Analysis of existing loss-sensitive tuning methods reveals that one reason these methods fail to further improve model performance is that they only utilize the output after the fully connected layers, ignoring the richer sample features before those layers. Visual comparisons of balanced and long-tail data models show that the poor performance of typical long-tail recognition models is largely due to the insufficient number of tail class samples, resulting in features extracted by the model lacking clear class boundaries. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide a long-tail visual recognition method, device and electronic device based on feature aggregation loss.
[0004] The content of this invention includes:
[0005] Training samples are obtained from long-tailed image datasets, and a long-tailed visual recognition model is trained based on the training samples;
[0006] During training, the long-tailed visual recognition model is first regularized to shrink the training samples in the feature space.
[0007] Then, influence factors are added to the long-tailed visual recognition model in turn to perform separation and aggregation operations on the training samples in the feature space. The separation operation is used to increase the boundary distance between features of different categories, and the aggregation operation is used to group features of the same category together.
[0008] The influence factors of the separation operation and the aggregation operation are then aggregated to obtain the feature aggregation loss. The long-tail visual recognition model is trained based on the feature aggregation loss to obtain the trained long-tail visual recognition model.
[0009] The trained long-tail visual recognition model is used to recognize the input image to be recognized.
[0010] According to the above-described technical solution of the present invention, the following improvements can also be made:
[0011] Optionally, the influencing factors include feature balancing factors and hard sample feature constraints. Before the training samples are separated and aggregated in the feature space, the original loss of the long-tail visual recognition model is determined. Then, a feature balancing factor is added to the original loss to obtain a new weighted loss so that the training samples are separated in the feature space. Finally, hard sample feature constraints are added to the new weighted loss so that the hard sample features in the training samples are aggregated in the feature space.
[0012] Optionally, the original loss is determined through the following steps:
[0013] Let the long-tail training samples be {x} i ;y i}, i∈{1,…,n j}, x k It refers to the i-th image sample belonging to category j in the training samples, y i It is the corresponding category label, n j Let P represent the number of samples in class j, j∈{1,…,C}, where C is the total number of classes in the samples, and P is the classification probability of each class in the training samples. i for:
[0014]
[0015] Where S represents the softmax function, P i ={P i1 ,P i2 ,…,P ij ,…,P iC},z ij =X i ω j X i ω represents the features extracted by the long-tailed visual recognition model. j If we refer to the classifier network associated with category j, then the original loss of the long-tail visual recognition model is expressed as:
[0016]
[0017] It is the one-hot encoded form of the sample label.
[0018] Optionally, the new weighted loss is determined by the following steps:
[0019] First, the probability distribution of the training samples is calculated using the following formula:
[0020]
[0021] This yields the probability distribution Ω = {R1, R2, ..., R} for each category. j , ..., RC Then, the calculated probability distribution is normalized using a minimum-maximum standardization:
[0022]
[0023] Then, the logit output of the long-tail visual recognition model is weighted:
[0024] Z′=(1+ψΩ′)Z T ;
[0025] Where 1+ψΩ′ is the feature balance factor, and ψ is the intensity of feature adjustment; Z T This is the output of the long-tailed visual recognition model;
[0026] Substituting the new weighted logit into the original loss yields the new weighted loss:
[0027]
[0028] Optionally, the hard sample feature constraint causes the tail class features to be distributed in the outermost layer of the space and to be sufficiently far from the head class features.
[0029] Optionally, hard sample feature constraints are determined through the following steps:
[0030] First, based on the features X in the training samples... i Calculate the class feature center P of the j-th class of the training samples. j :
[0031]
[0032] Then, based on the category feature center P i and P j Calculate the Euclidean distance between categories:
[0033] D ij =‖P i -P j ‖2;
[0034] By calculating the Euclidean distance between features and taking the maximum value of 0, we ensure that the hard sample feature constraint is always positive. Therefore, the hard sample feature constraint is expressed as:
[0035]
[0036] in, This represents an anchor sample selected from the tail class. This refers to positive samples selected from the tail class. This represents a negative sample selected from the head class. The distance between samples of the same type, Let α be the distance between features of different classes of samples, and α be the margin, where α = D. ij -average; average is the average distance between tail class features, and it is calculated as follows:
[0037]
[0038] n t X represents the number of tail classes in the training samples. k Fingertail class samples.
[0039] Optionally, the feature aggregation loss is defined as The feature aggregation loss is then expressed as:
[0040]
[0041] Optionally, the hard sample features are tail class features in the training samples.
[0042] According to another aspect of the present invention, a long-tail visual recognition device based on feature aggregation loss is provided, comprising:
[0043] The feature balancing factor module uses the number of class samples to adjust the output probability distribution of the visual recognition model, making the feature distance between different categories more balanced, thereby improving the classification accuracy of the visual recognition model.
[0044] The hard sample feature constraint module increases the boundary distance between different categories by performing cluster analysis on sample features, enabling the visual recognition model to find a more reasonable decision boundary.
[0045] According to another aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the long-tail visual recognition method based on feature aggregation loss as described above.
[0046] The beneficial effects of this invention are that it provides a long-tail visual recognition method, device, and electronic device based on feature aggregation loss. By performing shrinkage, separation, and aggregation operations on samples in the feature space, it enhances the recognition ability of difficult samples. Compared to the original cross-entropy loss, the class distribution boundaries of features in the space are clearer, thereby improving recognition accuracy. While maintaining the head class unchanged, it increases the feature norm of the tail class samples, thereby improving feature extraction capability and further enhancing recognition accuracy. Attached Figure Description
[0047] Figure 1 This is a schematic diagram of the long-tail distribution of the present invention.
[0048] Figure 2 This is a schematic diagram showing a visual comparison of the features of long-tail and balanced data in this invention.
[0049] Figure 3 This is a schematic diagram showing the comparison of the 2-norm of the weights of the category-based classifier of the present invention.
[0050] Figure 4 This is a schematic diagram illustrating the shrinkage, separation, and aggregation process of the sample in the feature space according to the present invention.
[0051] Figure 5 This is a schematic diagram of the difficult sample feature constraint process of the present invention.
[0052] Figure 6 This is a visual comparison diagram of the features of the present invention.
[0053] Figure 7 This is a schematic diagram comparing the feature norms of the present invention.
[0054] Figure 8 This is a schematic diagram illustrating the ablation experimental parameter analysis of the present invention. Detailed Implementation
[0055] As shown in the figure, the present invention includes the following steps:
[0056] Training samples are obtained from long-tailed image datasets, and a long-tailed visual recognition model is trained based on the training samples;
[0057] During training, the long-tailed visual recognition model is first regularized to shrink the training samples in the feature space.
[0058] Then, influence factors are added to the long-tailed visual recognition model in turn to perform separation and aggregation operations on the training samples in the feature space. The separation operation is used to increase the boundary distance between features of different categories, and the aggregation operation is used to group features of the same category together.
[0059] The influence factors of the separation operation and the aggregation operation are then aggregated to obtain the feature aggregation loss. The long-tail visual recognition model is trained based on the feature aggregation loss to obtain the trained long-tail visual recognition model.
[0060] The trained long-tail visual recognition model is used to recognize the input image to be recognized.
[0061] Unlike artificially created balanced datasets, natural data exhibits a severely imbalanced long-tail distribution, such as... Figure 1As shown, a few dominant categories (head classes) have a large number of samples, while many other categories (tail classes) have only a few samples. Because the tail class data samples are too few, visual recognition models based on long-tail data will overfit the tail class, making it difficult for long-tail visual recognition models to achieve satisfactory performance in tail class recognition. This greatly limits the application of deep models in the real world.
[0062] The technical solution of this embodiment mainly addresses two problems in long-tail recognition. First, weight decay is added to avoid model overfitting, which leads to the extracted feature norm being too small. Figure 7 As shown in the bottom line; the inability to effectively extract sample features in the test set limits the model's ability to represent tail classes. Secondly, models based on long-tail data have too few training samples, resulting in relatively scattered sample features. Figure 2 The right side; this makes it difficult for the model to find a suitable decision boundary during classification. Theoretical derivation revealed another important reason for the low accuracy of long-tail recognition: the weight bias in the model classifier caused by the different sample sizes in the training set, such as... Figure 3 As shown, visualizing the 2-norm of the model classifier weights reveals that the norm obtained by the baseline method BS training exhibits class bias, as shown below. Figure 3 As shown by the topmost line, the main issue is that the weight norm of the tail class is too small.
[0063] To address the above issues, this embodiment employs a feature aggregation loss, primarily comprising a feature balancing factor and hard sample feature constraints. By utilizing rich sample feature information, features are balanced and constrained, resulting in a more even distribution and better clustering of features in space. This embodiment adds two additional terms to the original loss: a feature balancing factor based on the number of class samples and feature constraints oriented towards hard samples. Figure 4 As shown, the feature aggregation loss adjusts features in three main stages. First, the addition of weight decay reduces overfitting in the visual recognition model and improves the recognition accuracy of samples represented by the head class. Second, by adding a feature balancing factor, features are distributed more evenly in the feature space, thereby increasing the boundary distance between features of different classes. Finally, with the help of the hard sample feature constraint module, features of the same class are clustered together, further increasing the class boundaries of features.
[0064] In this embodiment, h,m,t ∈ {1, ..., C}, then n can be used. h The number of times a head class is represented by n in the training set. m and n t Let n represent the number of mid-range and tail-range classes in the training set, respectively. Then, the long-tail problem can be represented by n. h >n m >n tThis means taking a sample {x} from the dataset. i ;y i Let F and W represent the feature extraction network and classifier network of the long-tail visual recognition model, respectively. Then, the process of extracting the features of this sample using this network can be represented as:
[0065]
[0066] in, X represents the parameters of the feature extraction network. i This represents the features extracted by the visual recognition model. Based on the above formula, the final output of the long-tail visual recognition model can be obtained:
[0067]
[0068] The parameters of the network classifier are represented as follows, where the classifier network can be represented as This refers to the classifier network associated with category j; therefore, Z = {z} i1 ;z i2 ;…;z ij ;…;z iC Therefore, the output of the long-tail visual recognition model regarding whether the sample belongs to the j-th category can be obtained:
[0069]
[0070] To improve the feature norm of tail class samples based on weight decay and enhance the model's feature extraction capability for tail class samples, a feature balancing factor is used to weight the visual recognition model output when calculating the loss. This newly added feature balancing factor promotes the growth of tail class features by providing more negative gradients, thereby adjusting the bias of the long-tail visual recognition model and making the features more evenly distributed in the feature space. Figure 4 As shown in (c).
[0071] The weights of the feature balancing factor are determined based on the number of samples in each category. First, the probability distribution of the training samples is calculated:
[0072]
[0073] This yields the probability distribution Ω = {R1, R2, ..., R} for each class. j , ..., R C Then, the calculated probability distribution is normalized using a minimum-maximum standardization:
[0074]
[0075] Then, the logit output of the long-tail visual recognition model is weighted, which provides more gradients for the head samples when calculating the final loss, thereby achieving a balance of features:
[0076] Z′=(1+ψΩ′)Z T ;
[0077] Where 1+ψΩ′ is the feature balancing factor, and ψ is the intensity of feature adjustment. A larger value will cause head-class features to aggregate better in the inner layer of the feature space, while tail-class features will be more biased towards the outermost layer of the feature space; Z T This is the output of the visual recognition model;
[0078] Substituting the new weighted logit into the original loss yields the new weighted loss:
[0079]
[0080] Based on the feature balancing factor's adjustment of features, the features at the current stage are now relatively evenly distributed across the feature space (e.g., Figure 4 (c) As shown, considering that optimizing and adjusting the boundaries for all samples would significantly increase the computational cost of the visual recognition model, and because most class features already exhibit good clustering in the long-tail problem, the categories requiring readjustment of feature class boundaries are mainly some difficult classes represented by the tail classes. Therefore, difficult samples are selected for aggregation during training to adjust the spatial clustering of sample features. Considering the significant differences in distance between categories in long-tail recognition, a difficult sample feature constraint based on class center distance is adopted. While maintaining the spatial layout of class features, this constraint further ensures that samples of the same class have similar feature performance, thereby improving the feature differences between samples of different classes, such as... Figure 5 As shown, Figure 5 The features of the tail classes identified on the left side of the middle label are not well clustered together; some features, such as It even intruded into the head class, causing the model to be unable to find a suitable classification boundary. Directly using triplet loss can lead to overfitting due to the similarity of tail class sample features. Therefore, adjusting the distance margins between features is extremely important. Hard sample feature constraints, by introducing the class feature center and the average distance of tail class samples, provide accurate distance margins for feature aggregation, ensuring generalization while aggregating similar features.
[0081] First, based on the features X in the long-tail training samples... i Calculate the class feature center P of the j-th class in the long-tail training samples. j :
[0082]
[0083] Then, based on the category feature center Pi and P j Calculate the Euclidean distance between categories:
[0084] D ij =‖P i -P j ‖2;
[0085] By calculating the Euclidean distance between features and taking the maximum value of 0, we ensure that the hard sample feature constraint is always positive. Therefore, the hard sample feature constraint is expressed as:
[0086]
[0087] Here, α represents the margin. The constraint on hard sample features requires maintaining the tail class features at the outermost layer of the spatial distribution, meaning the tail class has a larger feature norm, while simultaneously ensuring it is sufficiently far from the head class features. Based on weighted training using the feature balancing factor, the tail class features already possess a larger norm. Therefore, the margin setting should be consistent with the existing spatial distribution of class features, i.e., proportional to the distance between classes. Simultaneously, to avoid complete convergence of tail class features, the average distance between tail class features, α = D, needs to be subtracted from the class distance. ij -average; average is the average distance between tail class features. This represents an anchor sample selected from the tail class. This refers to positive samples selected from the tail class. This represents a negative sample selected from the head class. The distance between samples of the same type, Let α be the distance between features of different classes of samples, and α be the margin, where α = D. ij -average; average is the average distance between tail class features, and it is calculated as follows:
[0088]
[0089] n t X represents the number of tail classes in the training samples. k Fingertail class samples.
[0090] In this embodiment, the feature aggregation loss is defined as The feature aggregation loss is then expressed as:
[0091]
[0092] Experimental verification:
[0093] The experiment used three commonly used long-tail recognition datasets to verify the effectiveness of the overall method and its individual modules in the embodiment. This embodiment summarizes these three long-tail datasets in Table 1, and provides a detailed description below:
[0094] Table 1
[0095] Dataset Number of classes Training set samples Test set samples IF CIFAR-100-LT 100 10847 10000 100 ImageNet-LT 1000 115846 500000 256 iNat 2018 8142 437513 24426 435
[0096] The original CIFAR dataset contains 50,000 training images and 10,000 test images, all 32×32 3D color images. CIFAR10 is a coarse-grained version of this dataset, with 10 classes, 5,000 training images and 1,000 test images per class. CIFAR100 is a fine-grained version of the original dataset, dividing it into 100 classes, with 500 and 100 training and test images per class. Based on these two datasets, a long-tail version of the dataset was created. This version downsamples the training set exponentially by an imbalance factor (IF), while keeping the test set unchanged. The strength of the imbalance is controlled by IF, specifically calculated as IF = n... max / n min , where n max and n min These represent the maximum and minimum number of training images for each category. In this experiment, IF = {100; 50; 10} was used to complete all testing and validation. Based on the number of images in each category, they were divided into head, middle, and tail categories, with specific classification thresholds of 100 and 20.
[0097] ImageNet-LT is a long-tailed version of ImageNet, similar to the long-tailed version of the CIFAR dataset, with an imbalanced training set and a balanced test set. Like the original ImageNet-1K, it has 1,000 classes, and the training set is obtained by downsampling according to a Pareto distribution. The final training set contains 115,846 images, with the most populated class having 1,280 images and the least populated class having 5 images.
[0098] iNaturalist (iNat) is a real-world, fine-grained dataset for species classification and detection. It connects naturalists worldwide into a social network, generating high-quality biodiversity data through public participation, and then integrating this data into computer vision models to advance biodiversity awareness. The two most popular versions are iNat 2017, containing 579,184 training images across 5,089 categories, and version 2018, with 437,513 training samples across 8,142 categories. Its imbalance factor reaches 512. This example uses the latest version, iNaturalist 2018, to better simulate real-world scenarios.
[0099] Assessment and Basic Settings
[0100] For the basic training settings, all code was written using PyTorch on four GeForce GTX4090 GPUs. For the CIFAR100-LT dataset, ResNet-32 was used as the backbone network and linear classifier for all experiments, with a weight decay value of 4.E-03 that significantly impacted the experimental results. For the ImageNet-LT and iNaturalist2018 datasets, ResNet-50 and ResNeXt-50 were used as the backbone networks, respectively, with a weight decay value of 2.E-04. Other details, such as the learning rate, are listed in Table 2. The value of was set to 0.6, and its specific impact on model training will be further discussed in the ablation experiments. Top-1 accuracy was used to measure the model's recognition accuracy.
[0101] Table 2
[0102]
[0103]
[0104] Comparison of state-of-the-art methods:
[0105] This embodiment compares the proposed feature aggregation loss with several state-of-the-art methods on three datasets to verify its effectiveness. Specific comparison results are shown in Tables 3 and 4. This embodiment compares the proposed feature aggregation loss with other similar long-tail losses, multi-expert networks with higher space complexity, and state-of-the-art methods to verify the performance of the aggregation loss. The best results are shown in the last row of Table 3 and the second-to-last row of Table 4. The second-ranked results are 69.3 for SuperDisco in Table 3, and 59.8 and 56.3 for SHIKE; and 58.71 for the feature aggregation loss and 73.60 for SuperDisco in Table 4. The third-ranked results are 69.1 for PC in Table 3, and 58.3 and 53.8 for SuperDisco; and 57.10 for SuperDisco and 72.60 for the feature aggregation loss in Table 4.
[0106] Table 3
[0107]
[0108] Table 4
[0109] method source years ImageNet-LT iNat 2018 CE - - 41.60 61.00 CE-DRW NeurIPS 2019 50.10 - LDAM-DRW NeurIPS 2019 49.80 66.10 Causal Norm NeurIPS 2020 51.80 - BS NeurIPS 2020 51.00 70.00 Remix ECCV 2020 48.60 70.50 RIDE(3E) ICLR 2021 55.40 72.20 MiSLAS CVPR 2021 52.70 71.60 TSC CVPR 2022 54.20 69.70 WD CVPR 2022 53.90 70.20 KPS PAMI 2022 51.28 70.35 PC CVPR 2023 54.90 70.30 SuperDisco CVPR 2023 57.10 73.60 SHIKE CVPR 2023 59.70 74.50 Feature aggregation loss Example 2023 58.71 72.60
[0110] Table 3 shows the performance comparison of the proposed method on CIFAR-100-LT with imbalance factors of 100, 50, and 10. Compared with other similar methods, such as KPS, TSC, and LDAMRW, it achieves significant improvements at all imbalance rates. Comparing the feature aggregation loss with baseline methods (CE and BS), it can be found that the feature aggregation loss significantly improves the performance of CE and BS. For example, at an IF=100 setting, the feature aggregation loss improves the Top-1 accuracy of the baseline methods on CIFAR-100-LT by more than 19.6% and 7.4%, respectively. Compared with state-of-the-art methods in 2023, the proposed method still achieves competitive results, outperforming state-of-the-art methods at various imbalance factors (IF). Even more impressively, the end-to-end single-network performance of this embodiment surpasses that of more complex multi-stage and multi-expert networks. For example, it improves performance by 1.9% over the second-best method, SHIKE, on an imbalance factor of 100. Compared to these multi-expert networks, the feature aggregation loss of this embodiment adjusts the loss function based solely on the number of training samples, making it simpler to implement.
[0111] Table 4 shows the experimental results on two large-scale long-tail datasets. As shown in Table 4, the experimental results on the large-scale datasets are basically consistent with those on CIFAR100-LT. On these two large-scale datasets, it still outperforms the baseline and most state-of-the-art methods. On ImageNet-LT and iNaturalist 2018, the performance of the feature aggregation loss ranks second and third, respectively, still leading the recently proposed PC, KPS and other methods. Compared with the baseline method BS, the feature aggregation loss proposed in this embodiment improves the Top-1 accuracy on ImageNet-LT and iNaturalist 2018 by more than 7.71% and 1.9%, respectively, and the performance is significantly improved.
[0112] Feature Visualization Comparison
[0113] 1) Feature Visualization: To more intuitively understand the effect of the feature aggregation loss method, this embodiment selects 10 classes at equal intervals based on the CIFAR100-LT dataset and uses UMAP, which better preserves the global structure of the data, for visualization. The implementation process of the method is explained by showing the changes of features in space. For example... Figure 6 As shown, the left image is a comparison of training set samples, and the right image is a comparison of test set samples. The features extracted by the model using cross-entropy loss on the training set samples are scattered throughout the feature space, resulting in poor category clustering and making it difficult to find a reasonable decision classification boundary. In contrast, the feature aggregation loss proposed in this embodiment extracts features with good category characteristics, as shown in... Figure 4As illustrated in the method diagram, features not only cluster more tightly according to category, but also distribute across different layers of the feature space based on the difference in the number of samples per category, forming wider class boundaries and enabling more accurate decision boundaries. Figure 6 In the test set data on the right, the feature visualization results are consistent with those of the training dataset. The feature aggregation loss proposed in this embodiment, compared with the original cross-entropy loss, makes the class distribution boundaries of features in space clearer, thereby improving the recognition accuracy of the model.
[0114] 2) Feature Norm Comparison: Consistent with the previous section, we continue to use ten categories randomly selected from the CIFAR100-LT training set for visualization. For example... Figure 7 As shown, the lines at the top, bottom, and middle positions correspond to the cross-entropy loss, the weighted attenuation loss, and the feature aggregation loss proposed in this embodiment, respectively, which is a method illustration. Figure 4 The corresponding subgraphs b, c, and d show that under the original cross-entropy loss function, the feature norm extracted by the model is relatively large, leading to overfitting. After adding weight decay, the feature norm is significantly reduced, effectively solving the overfitting problem. However, the reduction in feature norm also limits the model's ability to extract features from tail-class samples, as... Figure 7 As shown in the middle line, the feature aggregation loss in this embodiment improves the feature norm of the tail class samples while keeping the head class unchanged, thereby improving the feature extraction capability of the model and further enhancing the recognition accuracy of the model.
[0115] Ablation experiment:
[0116] The parametric analysis of the weighted intensity ψ on the CIFAR100-LT dataset with an imbalance rate of 100 is shown in [link to relevant documentation]. Figure 8 As shown. To find a suitable value for ψ, experiments were conducted on the proposed feature aggregation loss with a series of ψ values. The strength of the feature weighting was controlled, and the best results were achieved when ψ = 0:6, indicating that the feature representation of the samples has reached a balanced state between the head and tail classes.
[0117] Ablation experiment of the module
[0118] To analyze the effectiveness and contribution weights of each module in the feature aggregation loss in detail, this embodiment further conducted a detailed ablation study on each module on the CIFAR100-LT dataset. The experimental results are shown in Table 5. To conduct a comprehensive analysis and avoid the influence of baseline method differences on the experiments, this embodiment used two baseline methods, CE and BS, for comparative experiments to more reasonably verify the effectiveness of the feature aggregation loss. The experiments mainly consist of... Figure 4The method comprises three components: weight decay, feature balancing factor, and hard sample feature constraint. Experimental results show that the proposed feature balancing factor improves performance by 1.6% and 0.6% on the CE and BS baseline methods, respectively. Further addition of the hard sample constraint results in a further significant performance improvement, with cumulative increases of 2.3% and 2.0%, respectively. Overall, all components proposed in this embodiment achieve continuous performance improvements across both baselines, demonstrating the effectiveness of the feature aggregation loss method used in this embodiment.
[0119] Table 5
[0120]
[0121]
[0122] Summarize
[0123] To address the bias in deep long-tail recognition models towards tail classes, this embodiment analyzes the causes of this phenomenon and proposes a feature aggregation loss method. This method not only expands the distance between classes but also aggregates features from samples of the same class. Higher recognition accuracy is achieved by adjusting the spatial distribution of features. The visualization results of the features are consistent with experimental expectations, showing that the proposed method achieves feature contraction, separation, and aggregation. Furthermore, experiments on multiple benchmark datasets demonstrate that the proposed method achieves state-of-the-art performance.
[0124] In another embodiment, a long-tail visual recognition device based on feature aggregation loss is provided, comprising:
[0125] The feature balancing factor module uses the number of class samples to adjust the output probability distribution of the model, making the feature distance between different classes more balanced, thereby improving the classification accuracy of the model.
[0126] The hard sample feature constraint module increases the boundary distance between different categories by performing cluster analysis on sample features, enabling the model to find a more reasonable decision boundary.
[0127] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the long-tail visual recognition method based on feature aggregation loss as described in any of the above embodiments.
[0128] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.
[0129] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.
Claims
1. A long-tail visual recognition method based on feature aggregation loss, characterized by: It includes the following steps: Training samples are obtained from long-tailed image datasets, and a long-tailed visual recognition model is trained based on the training samples; During training, the long-tailed visual recognition model is first regularized to shrink the training samples in the feature space. Then, influence factors are added sequentially to the long-tail visual recognition model to perform separation and aggregation operations on the training samples in the feature space. The separation operation is used to increase the boundary distance between features of different categories, and the aggregation operation is used to group features of the same category together. The influence factors include feature balancing factors and hard sample feature constraints. Before performing separation and aggregation operations on the training samples in the feature space, the original loss of the long-tail visual recognition model is determined. Then, a feature balancing factor is added to the original loss to obtain a new weighted loss so that the training samples can perform separation operations in the feature space. Then, hard sample feature constraints are added to the new weighted loss so that the hard sample features in the training samples can perform aggregation operations in the feature space. The new weighted loss is determined through the following steps: First, the probability distribution of the training samples is calculated using the following formula: ; This yields the probability distribution for each category. Then, the calculated probability distribution is normalized to minimum and maximum: ; Then, the logit output of the long-tail visual recognition model is weighted: ; in, As a characteristic balance factor, The intensity of the feature adjustment; This is the output of the long-tailed visual recognition model; Substituting the new weighted logit into the original loss yields the new weighted loss: ; Hard sample feature constraints are determined through the following steps: First, based on the features in the training samples Calculate the training sample number Class feature center : ; Then based on the category feature center and Calculate the Euclidean distance between categories: By calculating the Euclidean distance between features and taking the maximum value of 0, we ensure that the hard sample feature constraint is always positive. Therefore, the hard sample feature constraint is expressed as: ; in, This represents an anchor sample selected from the tail class. This refers to positive samples selected from the tail class. This represents a negative sample selected from the head class. The distance between samples of the same type, The distance between features of different classes of samples. For the edge, ; The average distance between tail class features is calculated as follows: , n t This indicates the number of tail classes in the training samples. Fingertail class samples; The influence factors of the separation operation and the aggregation operation are then aggregated to obtain the feature aggregation loss. The long-tail visual recognition model is trained based on the feature aggregation loss to obtain the trained long-tail visual recognition model. The trained long-tail visual recognition model is used to recognize the input image to be recognized.
2. The long-tail visual recognition method based on feature aggregation loss as described in claim 1, characterized in that, the original... The loss is determined through the following steps: Set the long-tail training samples as , {1,…, }, It refers to the i-th image sample belonging to category j in the training samples. These are the corresponding category labels. Represents the number of samples in class j, j {1, ..., C}, where C is the total number of categories in the sample and the classification probability of each category in the training samples. for: , Where S represents the softmax function. , = , This represents the features extracted by the long-tail visual recognition model. If we refer to the classifier network associated with category j, then the original loss of the long-tail visual recognition model is expressed as: ; It is the one-hot encoded form of the sample label.
3. The long-tail visual recognition method based on feature aggregation loss as described in claim 1, characterized in that, The hard sample feature constraint ensures that the tail class features are distributed in the outermost layer of the space and that the tail class features are sufficiently far away from the head class features.
4. The long-tail visual recognition method based on feature aggregation loss as described in claim 1, characterized in that, a definition is provided. Feature aggregation loss is The feature aggregation loss is then expressed as: 。 5. The long-tail visual recognition method based on feature aggregation loss as described in claim 1, characterized in that, The features of the difficult samples are the tail class features in the training samples.
6. A long-tail visual recognition device based on feature aggregation loss, characterized in that, The apparatus is configured to perform the long-tailed visual recognition method based on feature aggregation loss as described in any one of claims 1 to 5.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the long-tail visual recognition method based on feature aggregation loss as described in any one of claims 1 to 5.
