Long tail image recognition method based on hierarchical expert logit fusion

Abstract

This invention discloses a long-tail image recognition method based on hierarchical expert log-odds fusion. This method achieves accurate recognition of long-tail images using a hierarchical expert neural network. Its structure includes: a dual-path hierarchical expert sub-network consisting of independent expert paths and shared-decoupled expert paths. The decoupled experts have lightweight parameters based on task complexity scores and incorporate the knowledge of independent experts through a cross-expert feature fusion mechanism. During the training phase, a category routing module hierarchically assigns samples to different experts according to the "category-expert mapping" relationship. During the application phase, samples are assigned to all experts to obtain the log-odds of each expert. The log-odds fusion module aggregates the log-odds within the category intervals focused on by each expert and introduces a coordination correction term based on the output distribution differences to correct for errors, ultimately generating the prediction result. This invention can improve expert diversity, coordinate differences in multi-expert outputs, and enhance the recognition performance and robustness of long-tail data.

Description

Technical Field

[0001] This invention relates to the technical field of computer vision and deep learning image classification, and in particular to a long-tail image recognition method based on hierarchical expert log probability fusion. Background Technology

[0002] With the development of deep learning, convolutional neural networks have achieved remarkable results in the field of computer vision. Architectures such as residual networks effectively alleviate the vanishing gradient problem in deep networks through residual connections, improving model performance. However, the success of these models depends on large-scale and class-balanced datasets. In real-world scenarios such as industrial defect detection and medical image analysis, data often exhibits a "long-tail distribution": a few head classes have abundant samples, while most tail classes have scarce samples. This imbalance causes the model to be biased towards head classes during training, making it difficult to fully learn tail features, thus severely impacting overall classification performance, especially the accuracy of tail class recognition.

[0003] To address this issue, existing research mainly employs rebalancing methods and decoupling training strategies. The former suffers from the drawback of resampling leading to overfitting of tail samples, while reweighting can cause training instability. The latter, decoupling, involves two-stage training, a complex process that depends on the quality of feature extraction and is difficult to deploy end-to-end.

[0004] Multi-expert models, by integrating sub-network modules, combine the strengths of different experts to improve the recognition ability of tail classes while maintaining performance in the head classes, demonstrating good generalization potential. The core idea lies in leveraging the complementarity among experts to enhance the model's sensitivity to rare classes through collaborative decision-making. However, the practical effectiveness of such methods is limited by two key factors: First, there is insufficient diversity among experts. Existing methods often introduce differentiation mechanisms at the classifier or loss function level, which can create differentiation in output decisions, but their feature learning is still based on a shared, biased representation space influenced by long-tail distribution, leading to convergence of essential features among experts. In addition, the common practice of sharing some parameters to control model complexity further exacerbates the homogenization of the feature extraction process and limits true functional division of labor.

[0005] Second, there are significant semantic and distributional differences in the outputs of various experts. Due to differences in optimization paths and sample coverage during training, the output characteristics of each expert, such as confidence distribution and category response patterns, are often inconsistent, resulting in a lack of semantic alignment in their prediction results. This inconsistency in output distribution makes it difficult for simple ensemble to effectively integrate the knowledge of various experts, and may instead introduce conflicts or noise, hindering the improvement of overall performance. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a long-tail image recognition method based on hierarchical expert log-probability fusion. This method can effectively enrich the diversity of expert models, unify and coordinate the differential semantic output of multiple experts, significantly improve the effectiveness of the integration framework in long-tail learning, realize an end-to-end efficient integrated multi-expert network, and thus comprehensively improve the overall recognition accuracy and tail category classification performance of the neural network under long-tail distribution.

[0007] To achieve the above objectives, the technical solution provided by this invention is: a long-tail image recognition method based on hierarchical expert log-odds fusion. This method achieves accurate recognition of long-tail images based on a hierarchical expert neural network. The neural network includes a dual-path hierarchical expert sub-network, a category routing module, and a log-odds fusion module. The dual-path hierarchical expert sub-network consists of independent expert paths and shared-decoupled expert paths. The independent expert paths are constructed using three residual convolutional modules. The shared-decoupled expert paths are constructed using one shared network and two decoupled branch networks, resulting in two decoupled experts, namely decoupled expert 1 and decoupled expert 2. The shared network is a residual convolutional module, and each of the two decoupled branch networks consists of two residual convolutional modules. The decoupled branch network is parameter-lightweighted based on task complexity scores, and a cross-expert feature fusion mechanism is introduced to supplement the knowledge of independent experts to the decoupled experts, thereby enriching the features of the decoupled experts. The category routing module uses a two-level routing mechanism during the training phase to assign samples of different categories to corresponding experts according to the "category-expert mapping" relationship, so that experts can focus on different category intervals and improve the diversity of experts. In the application phase, the samples are assigned to all experts to obtain the log odds of the outputs of all experts. The log odds fusion module aggregates the log odds of the outputs based on the category intervals processed by the experts, and calculates a coordination correction term based on the differences in output distribution to unify and coordinate the differential semantic outputs of different experts, significantly improving the consistency and robustness of the neural network. The specific implementation of the long-tail image recognition method includes: Input the long-tailed image to be tested into the trained hierarchical expert-based neural network and perform the following operations: First, long-tailed images are assigned to a dual-path hierarchical expert subnetwork via a category routing module, where the dual-path hierarchical expert subnetwork has been parameter-lightweighted based on task complexity scores. In the independent expert path, long-tail images are feature extracted by independent experts. The second residual convolutional module of the independent expert path generates the intermediate features of the independent experts. In the shared-decoupled expert path, shallow features are first extracted through a shared network, and then features are extracted by two decoupled experts respectively. The first residual convolutional module of the decoupled branch network generates the intermediate features of decoupled expert 1 and decoupled expert 2 respectively. In the feature extraction process, in order to promote knowledge complementarity among experts and enhance the feature representation capability of the shared-decoupled expert path, the mid-level features of the extracted independent experts are fused into the mid-level features of decoupled experts 1 and 2 through a cross-expert feature fusion mechanism. Finally, the long-tailed image is processed by the last residual convolution module of the three experts to generate their respective deep features, which are then output to the fully connected layer of each expert. The original logical output of the neural network, i.e., log odds, is then obtained. Finally, with the help of the log odds fusion module, the log odds are aggregated in the category intervals focused on by each expert. Then, a coordination correction term based on the output distribution difference is introduced to correct the aggregated result. Finally, the prediction result of the long-tail image category is generated by softmax.

[0008] Furthermore, the "category-expert mapping" relationship in the category routing module is as follows: Independent Expert: ; Decoupling Expert 1: ; Decoupling Expert 2: ; In the formula, , , These represent the subsets of datasets received by the independent expert, decoupling expert 1, and decoupling expert 2, respectively. Represents sample-label pairs, This represents image samples from long-tail data. This represents the category label of image samples in long-tail data. This represents the head category range of long-tail data. This represents the middle category interval of long-tail data. This represents the tail category interval of long-tail data.

[0009] Furthermore, the category routing module performs the following operations: First, all category labels in the long-tail data are sorted in descending order based on the number of samples. Then, following the principle of equal distribution of category numbers, it is divided into three intervals with equal number of categories: the head category interval, the middle category interval, and the tail category interval. Next, hierarchical data allocation is performed based on the "category-expert mapping" relationship, dividing the originally highly imbalanced long-tail data into a less imbalanced subset. This allows different experts to focus on categories with varying data richness. Specifically: independent experts handle all categories, decoupled expert 1 handles the middle and tail categories, and decoupled expert 2 handles the tail category. Furthermore, the two-level routing mechanism of the category routing module is as follows: Level 1: Independent Expert Paths and Sharing - Decoupling expert path data allocation to ensure that independent experts receive a subset of the dataset. The samples enter the independent expert path, decoupling the dataset subset received by expert 1. The samples enter the shared-decoupling expert path; Level 2: Decoupling expert data allocation. After the samples extract shallow features in the shared network of the shared-decoupling expert path, ensure that the datasets received by Decoupling Expert 1 and Decoupling Expert 2 are subsets of the datasets. , The samples were then sent to the respective experts.

[0010] Furthermore, parameter lightweighting is performed based on task complexity scores, as follows: In classification tasks, the number of samples and the number of classes in long-tailed images both reflect the complexity of the task. After hierarchical data allocation, the dataset subsets processed by each expert differ in sample distribution and class range. To quantify the complexity of the classification tasks undertaken by different experts, a task complexity score is introduced as the basis for parameter lightweighting: Let the number of samples in the dataset subset processed by each expert be... The number of categories is The average number of samples per class is defined as Task complexity score The calculation formula is: ; In the formula, Represents the logarithmic function with base 2. , Let be the weighting coefficient, satisfying and The conditions are set to reflect the more dominant influence of the average number of samples in each class on the capacity of the neural network; Score based on the task complexity of independent experts As a benchmark score : ; In the formula, , These represent the number of samples and the number of categories in the subset of the dataset processed by independent experts, respectively. Let the task complexity score of decoupling expert e be... The relative task complexity score ratio for: ; In the formula, e = 1, 2; the above ratio This reflects the ratio of the required neural network capacity of the decoupled expert e to that required by the independent expert; if This indicates that the decoupling expert e has low task complexity and can lightweight its network parameters; based on The number of channels in the last two residual convolutional modules of the decoupling expert e is scaled according to the following rules: ; In the formula, The number of channels for decoupling expert e. The number of channels for independent experts. The expression indicates taking the nearest integer. "Adjust to even number()" indicates the rounding strategy, that is: if the rounded number is still odd, then take the nearest even number to ensure that the capacity is not excessively reduced.

[0011] Furthermore, to promote knowledge complementarity among experts and enhance the feature representation capabilities of shared-decoupled expert paths, the cross-expert feature fusion mechanism integrates the mid-level features of independent experts into the mid-level features of decoupled experts, as detailed below: The feature fusion process of Decoupling Expert 1 is as follows: Extracting the mid-level features of independent experts sequentially Mid-level features of decoupling expert 1 Align the channel count and width / height of the mid-layer features, then concatenate them. Next, perform convolutional fusion on the concatenated feature maps to obtain the fused features of Decoupling Expert 1. Finally, batch normalization and ReLU activation are performed to obtain the enhanced features of decoupling expert 1. , is represented as: ; ; The feature fusion process of Decoupling Expert 2 is as follows: Extracting mid-level features from independent experts sequentially Mid-layer features of Decoupling Expert 2 Align the channel count and width / height of the mid-layer features, then concatenate them. Next, perform convolutional fusion on the concatenated feature maps to obtain the fused features of Decoupling Expert 2. Finally, batch normalization and ReLU activation are performed to obtain the enhanced features of Decoupling Expert 2. , is represented as: ; ; In the formula, This represents a 1×1 convolution operation. This indicates feature alignment and splicing operations. This indicates batch normalization operation.

[0012] Furthermore, preliminary fusion is performed within the category intervals focused on by each expert, specifically by performing the following calculations: For the top category intervals, only independent experts process the data, and the logarithmic probability of each independent expert is used as the output. For the middle category interval, both the independent expert and decoupling expert 1 are processed, and the log-probability outputs for both are provided. , Take the sum and average, that is For the tail category interval, output the logarithmic odds of independent expert, decoupling expert 1, and decoupling expert 2. , and Take the sum and average, that is The preliminary fusion results were obtained through the above calculations.

[0013] Furthermore, a reconciliation correction term based on output distribution differences is introduced to correct the preliminary fusion results. The principle is based on the Jensen-Shannon divergence, which measures the degree of disagreement among experts at the probability distribution output level. A reconciliation correction term is designed to adjust the preliminary fusion results, as follows: Probability distribution transformation and mean distribution calculation: Input the long-tailed image to be tested, obtain the log odds for each expert, first transform each expert... Log odds on category c Convert to probability distribution And calculate the average distribution. : ; In the formula, This represents the number of experts with log-probability outputs in category c, with values ​​of 1, 2, or 3. This represents the expert identifier, with values ​​1, 2, and 3, corresponding to independent expert, decoupling expert 1, and decoupling expert 2, respectively. Normalized distribution difference calculation: Based on the Jensen-Shannon divergence, the degree of difference between the expert output distribution and the mean distribution is calculated and normalized. ; In the formula, Indicates the degree of difference in the distribution after normalization. This represents a logarithmic function with the natural constant as its base. express The process of calculating divergence, The larger the value, the greater the semantic discrepancy among experts; Reconciliation Correction and Final Output: Utilizing the calculated degree of distributional difference, the log-odds ratio after initial fusion is adjusted. After reconciliation and correction, the final log-odds output is generated. : ; In the formula, To correct the intensity coefficient, the greater the semantic output discrepancy among experts, the less reliable the preliminary fusion result is. Therefore, by using a coordination correction term, the log probability result of the preliminary fusion is adaptively reduced, and then the probability distribution output is reduced after softmax normalization to lower the confidence for category c.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. Enhancing the diversity among experts and alleviating the problem of homogenized feature representation: This invention assigns data with different distribution characteristics to corresponding experts through a designed category routing module, enabling each expert to focus on feature learning within a specific category interval, thereby forming differentiated expert performance. Furthermore, the dual-path hierarchical expert sub-network proposed in this invention abandons the traditional mechanism of multiple experts sharing some parameters. The constructed independent expert path and shared-decoupled expert path adopt different network structures and parameter configurations. Combined with the cross-expert feature fusion mechanism, it effectively alleviates the problem of homogenized feature representation in traditional multi-expert models and enriches the model's representational capabilities.

[0015] 2. Achieves efficient end-to-end integration and semantic alignment of outputs, enhancing the consistency of multi-expert prediction results: The log-odds fusion module proposed in this invention solves the problem of dimensional misalignment of multi-expert outputs by aggregating the effective output intervals of experts. Furthermore, it introduces a coordination correction term based on the difference in output information, calculates the statistical difference between the distributions of each expert's output, and corrects the preliminary fusion results, making the model outputs tend to align in the semantic space, thereby improving the consistency and stability of the multi-expert system's prediction results for the same sample.

[0016] 3. Controlling neural network complexity and improving overall recognition performance under long-tail data: By designing a neural network structure based on hierarchical experts and lightweighting parameters based on task complexity scores, this invention achieves optimized allocation of computing resources while controlling neural network complexity, maintaining high-level majority class recognition performance while also improving the classification accuracy of minority classes. Attached Figure Description

[0017] Figure 1This is a structural diagram of a neural network based on hierarchical experts. In the diagram, the first-level category routing module inputs samples from all categories into independent expert paths and samples from the middle and tail categories into shared decoupled expert paths. The second-level category routing module inputs samples from the middle and tail categories into decoupled expert 1 and samples from the tail category into decoupled expert 2. ResBlock represents a residual convolution module, and ResBlock-Lightweight represents a residual convolution module with lightweight parameters. Feature fusion represents the fusion of the middle-layer features of independent experts into the middle-layer features of decoupled experts 1 and 2 across experts. Linear represents a fully connected layer that transforms the feature representation into an unnormalized logical output, i.e., log odds. The log odds fusion module fuses the logical outputs of the experts to output the final prediction result. Detailed Implementation

[0018] The present invention will be further described below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0019] This embodiment discloses a long-tail image recognition method based on hierarchical expert log-odds fusion. This method achieves accurate recognition of long-tail images using a hierarchical expert neural network. The neural network includes a dual-path hierarchical expert subnetwork, a category routing module, and a log-odds fusion module. The dual-path hierarchical expert subnetwork consists of independent expert paths and shared-decoupled expert paths. The independent expert paths are constructed using three residual convolutional modules. The shared-decoupled expert paths are constructed using one shared network and two decoupled branch networks, resulting in two decoupled experts, namely decoupled expert 1 and decoupled expert 2. The shared network is a residual convolutional module, and each of the two decoupled branch networks consists of two residual convolutional modules. The method is based on task complexity... The noise score is used to lightweight the parameters of the decoupled branch network. A cross-expert feature fusion mechanism is introduced to supplement the knowledge of independent experts to the decoupled experts, enriching their features. During the training phase, the category routing module uses a two-level routing mechanism to assign samples of different categories to corresponding experts according to the "category-expert mapping" relationship, allowing experts to focus on different category intervals and improving expert diversity. In the application phase, samples are assigned to all experts, obtaining the log-odds of all expert outputs. The log-odds fusion module aggregates the log-odds of the outputs based on the category intervals processed by the experts and calculates a coordination correction term based on the differences in output distribution to unify and coordinate the differential semantic outputs of different experts, significantly improving the consistency and robustness of the neural network.

[0020] The specific implementation of the long-tail image recognition method includes: First, prepare long-tail data using the public datasets CIFAR100-LT and CIFAR10-LT series. Common imbalance rates include 10, 50, and 100. Use long-tail datasets with different balance rates to measure the predictive ability of the neural network under different data biases.

[0021] Next, the training of the hierarchical expert-based neural network was carried out, using ResNet32 as the backbone. During the training phase, the "category-expert mapping" relationship in the category routing module was as follows: Independent Expert: ; Decoupling Expert 1: ; Decoupling Expert 2: ; In the formula, , , These represent the subsets of datasets received by the independent expert, decoupling expert 1, and decoupling expert 2, respectively. Represents sample-label pairs, This represents image samples from long-tail data. This represents the category label of image samples in long-tail data. This represents the head category range of long-tail data. This represents the middle category interval of long-tail data. This represents the tail category interval of long-tail data.

[0022] The category routing module first sorts all category labels in the long-tail data in descending order based on the number of samples. Then, following the principle of equal distribution of category numbers, it divides the data into three intervals with equal number of categories: the head category interval, the middle category interval, and the tail category interval. Next, based on the "category-expert mapping" relationship, hierarchical data allocation is performed, dividing the originally highly imbalanced long-tail data into subsets with lower imbalance. This allows different experts to focus on categories with varying data richness. Specifically, an independent expert handles all categories, decoupled expert 1 handles the middle and tail categories, and decoupled expert 2 handles the tail category. Through hierarchical data allocation, each expert develops a performance advantage in the head, middle, and tail category intervals, respectively.

[0023] The two-level routing mechanism of the category routing module is as follows: Level 1: Independent Expert Paths and Sharing - Decoupling expert path data allocation to ensure that independent experts receive a subset of the dataset. The samples enter the independent expert path, decoupling the dataset subset received by expert 1. The samples enter the shared-decoupling expert path.

[0024] Level 2: Decoupling expert data allocation. After the samples extract shallow features in the shared network of the shared-decoupling expert path, ensure that the datasets received by Decoupling Expert 1 and Decoupling Expert 2 are subsets of the datasets. , The samples were then sent to the respective experts.

[0025] After training the neural network, the long-tailed image to be tested is input into the trained hierarchical expert-based neural network (network architecture as follows). Figure 1 (As shown) Perform the following operations: S1. Long-tailed images are assigned to a dual-path hierarchical expert subnetwork by the category routing module. The dual-path hierarchical expert subnetwork has been parameter-lightweighted based on task complexity scores. The specific details of parameter lightweighting for the task complexity score are as follows: In classification tasks, the number of samples and the number of classes in long-tailed images both reflect the task's complexity. After hierarchical data allocation, the subset of datasets processed by each expert differs in sample distribution and class range. To quantify the complexity of the classification tasks undertaken by different experts, a task complexity score is introduced as a basis for parameter lightweighting: Let the number of samples in the subset of the dataset processed by the expert be... The number of categories is The average number of samples per class is defined as Task complexity score The calculation formula is: ; in, Represents the logarithmic function with base 2. , Let be the weighting coefficient, satisfying and The conditions are set to reflect the more dominant influence of the average number of samples per class on the capacity of the neural network.

[0026] Furthermore, the task complexity score is based on independent experts. As a benchmark score : ; In the formula, , These represent the number of samples and the number of categories in the subset of the dataset processed by independent experts, respectively. Let the task complexity score of decoupling expert e be... The relative task complexity score ratio for: ; In the formula, e = 1, 2; the above ratio This reflects the ratio of the required neural network capacity of the decoupled expert e to that required by the independent expert; if This indicates that the decoupling expert e has low task complexity and can lightweight its network parameters; based on The number of channels in the last two residual convolutional modules of the decoupling expert e is scaled according to the following rules: ; In the formula, The number of channels for decoupling expert e. The number of channels for independent experts. The expression indicates taking the nearest integer. "Adjust to even number()" indicates the rounding strategy, that is: if the rounded number is still odd, then take the nearest even number to ensure that the capacity is not excessively reduced.

[0027] S2. Data enters the dual-path hierarchical expert sub-network. In the independent expert path, the long-tail image is feature extracted by the independent expert. The second residual convolution module of the independent expert path generates the intermediate features of the independent expert. In the shared-decoupled expert path, shallow features are first extracted through the shared network, and then features are extracted by the two decoupled experts respectively. The first residual convolution module of the decoupled branch network generates the intermediate features of decoupled expert 1 and decoupled expert 2 respectively.

[0028] S3. In the feature extraction process, in order to promote knowledge complementarity among experts and enhance the feature representation capability of the shared-decoupled expert path, the extracted independent expert mid-level features are fused into the mid-level features of decoupled experts 1 and 2 through a cross-expert feature fusion mechanism. Finally, the long-tailed image is processed by the last residual convolution module of the three experts to generate their respective deep features, and output to the fully connected layer of each expert. Then, the original logical output of the neural network, i.e., log odds, is obtained. The cross-expert feature fusion mechanism integrates the mid-level features of independent experts into the mid-level features of decoupled experts, as detailed below: The feature fusion process of Decoupling Expert 1 is as follows: Extracting mid-level features from independent experts sequentially Mid-level features of decoupling expert 1 Align the channel count and width / height of the mid-layer features, then concatenate them. Next, perform convolutional fusion on the concatenated feature maps to obtain the fused features of Decoupling Expert 1. Finally, batch normalization and ReLU activation are performed to obtain the enhanced features of decoupling expert 1. , is represented as: ; ; The feature fusion process of Decoupling Expert 2 is as follows: Extracting mid-level features from independent experts sequentially Mid-layer features of Decoupling Expert 2 Align the channel count and width / height of the mid-layer features, then concatenate them. Next, perform convolutional fusion on the concatenated feature maps to obtain the fused features of Decoupling Expert 2. Finally, batch normalization and ReLU activation are performed to obtain the enhanced features of Decoupling Expert 2. , is represented as: ; ; In the formula, This represents a 1×1 convolution operation. This indicates feature alignment and splicing operations. This indicates batch normalization operation.

[0029] S4. After obtaining the log odds of each expert, the log odds are aggregated in the category intervals that each expert focuses on under the action of the log odds fusion module. Then, a coordination correction term based on the difference in output distribution is introduced to correct the aggregated result. Finally, the prediction result of the long-tail image category is generated through softmax.

[0030] Preliminary fusion is performed within the category intervals focused on by each expert, specifically by executing the following calculations: For the top category intervals, only independent experts are processed, and the logarithmic probability output of the independent experts is used. For the middle category interval, both the independent expert and decoupling expert 1 are processed, and the log-probability outputs for both are provided. , Take the sum and average, that is For the tail category interval, output the logarithmic odds of independent expert, decoupling expert 1, and decoupling expert 2. , and Take the sum and average, that is The preliminary fusion results were obtained through the above calculations.

[0031] After obtaining the initial fusion results, a reconciliation correction term based on the output distribution differences is used to correct the aggregated results. The principle is based on the Jensen-Shannon divergence, which measures the degree of disagreement among experts at the probability distribution output level. A reconciliation correction term is designed to adjust the initial fusion results, as follows: Probability distribution transformation and mean distribution calculation: Input the long-tailed image to be tested, obtain the log odds for each expert, first transform each expert... Log odds on category c Convert to probability distribution And calculate the average distribution. : ; In the formula, This represents the number of experts with log-probability outputs in category c, with values ​​of 1, 2, or 3. This represents the expert identifier, with values ​​1, 2, and 3, corresponding to independent expert, decoupling expert 1, and decoupling expert 2, respectively. Normalized distribution difference calculation: Based on the Jensen-Shannon divergence, the degree of difference between the expert output distribution and the mean distribution is calculated and normalized. ; In the formula, Indicates the degree of difference in the distribution after normalization. This represents a logarithmic function with the natural constant as its base. express The process of calculating divergence, The larger the value, the greater the semantic discrepancy among experts; Reconciliation Correction and Final Output: Utilizing the calculated degree of distributional difference, the log-odds ratio after initial fusion is adjusted. After reconciliation and correction, the final log-odds output is generated. : ; In the formula, To correct the intensity coefficient, the greater the semantic output discrepancy among experts, the less reliable the preliminary fusion result is. Therefore, by using a coordination correction term, the log probability result of the preliminary fusion is adaptively reduced, and then the probability distribution output is reduced after softmax normalization to lower the confidence for category c.

[0032] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A long tail image recognition method based on hierarchical expert pair number probability fusion, characterized in that, This method uses a hierarchical expert neural network to recognize long-tailed images. The neural network includes a dual-path hierarchical expert subnetwork, a category routing module, and a log-odds fusion module. The dual-path hierarchical expert subnetwork consists of independent expert paths and shared-decoupled expert paths. The independent expert paths are constructed using three residual convolutional modules. The shared-decoupled expert paths are constructed using a shared network and two decoupled branch networks, resulting in two decoupled experts, designated as Decoupled Expert 1 and Decoupled Expert 2. The shared network consists of one residual convolutional module, and each of the two decoupled branch networks consists of two residual convolutional modules. The method is task-based. Complexity scoring is used to lightweight the parameters of the decoupled branch network, while a cross-expert feature fusion mechanism is introduced to supplement the knowledge of independent experts to the decoupled experts. The category routing module uses a two-level routing mechanism during the training phase to assign samples of different categories to corresponding experts according to the "category-expert mapping" relationship, so that experts can focus on different category intervals. In the application phase, the samples are assigned to all experts to obtain the log odds of the outputs of all experts. The log odds fusion module aggregates the log odds of the outputs based on the category intervals processed by the experts, and calculates a coordination correction term based on the differences in output distribution to unify and coordinate the differential semantic outputs of different experts. The specific implementation of the long-tail image recognition method includes: Input the long-tailed image to be tested into the trained hierarchical expert-based neural network and perform the following operations: First, long-tailed images are assigned to a dual-path hierarchical expert subnetwork via a category routing module, where the dual-path hierarchical expert subnetwork has been parameter-lightweighted based on task complexity scores. In the independent expert path, long-tail images are feature extracted by independent experts. The second residual convolutional module of the independent expert path generates the intermediate features of the independent experts. In the shared-decoupled expert path, shallow features are first extracted through a shared network, and then features are extracted by two decoupled experts respectively. The first residual convolutional module of the decoupled branch network generates the intermediate features of decoupled expert 1 and decoupled expert 2 respectively. In the feature extraction process, in order to promote knowledge complementarity among experts and enhance the feature representation capability of the shared-decoupled expert path, the mid-level features of the extracted independent experts are fused into the mid-level features of decoupled experts 1 and 2 through a cross-expert feature fusion mechanism. Finally, the long-tailed image is processed by the last residual convolution module of the three experts to generate their respective deep features, which are then output to the fully connected layer of each expert. The original logical output of the neural network, i.e., log odds, is then obtained. Finally, with the help of the log odds fusion module, the log odds are aggregated in the category intervals focused on by each expert. Then, a coordination correction term based on the output distribution difference is introduced to correct the aggregated result. Finally, the prediction result of the long-tail image category is generated by softmax. 2.The long-tail image recognition method based on hierarchical expert pair number probability fusion according to claim 1, wherein, The "category-expert mapping" relationship in the category routing module is as follows: Independent Expert: ; Decoupling Expert 1: ; Decoupling Expert 2: ; In the formula, , , These represent the subsets of datasets received by the independent expert, decoupling expert 1, and decoupling expert 2, respectively. Represents sample-label pairs, This represents image samples from long-tail data. This represents the category label of image samples in long-tail data. This represents the head category range of long-tail data. This represents the middle category interval of long-tail data. This represents the tail category interval of long-tail data. 3.The long-tail image recognition method based on hierarchical expert pair number likelihood fusion according to claim 2, characterized in that, The category routing module performs the following operations: First, sort all category labels in the long-tail data in descending order according to the number of samples. Then, according to the principle of equal distribution of the number of categories, divide it into three intervals with equal number of categories: the head category interval, the middle category interval, and the tail category interval. Then, hierarchical data allocation is performed based on the "category-expert mapping" relationship, dividing the original long-tail data with high imbalance into a subset of datasets with low imbalance, so that different experts can focus on categories with different data richness. Specifically, independent experts handle all categories, decoupled expert 1 handles the middle and tail categories, and decoupled expert 2 handles the tail category. Through hierarchical data allocation, each expert forms a performance advantage in the head category interval, the middle category interval, and the tail category interval, respectively.

4. The long-tail image recognition method based on hierarchical expert pair number of logarithmic probability fusion according to claim 3, characterized in that, The two-level routing mechanism of the category routing module is as follows: Level 1: Independent Expert Paths and Sharing - Decoupling expert path data allocation to ensure that independent experts receive a subset of the dataset. The samples enter the independent expert path, decoupling the dataset subset received by expert 1. The samples enter the shared-decoupling expert path; Level 2: Decoupling expert data allocation. After the samples extract shallow features in the shared network of the shared-decoupling expert path, ensure that the datasets received by Decoupling Expert 1 and Decoupling Expert 2 are subsets of the datasets. , The samples were then sent to the respective experts.

5. The long-tail image recognition method based on hierarchical expert pair number of logarithmic probability fusion according to claim 4, characterized in that, Parameter lightweighting is performed based on task complexity scores, as follows: In classification tasks, the number of samples and the number of classes in long-tailed images both reflect the complexity of the task. After hierarchical data allocation, the dataset subsets processed by each expert differ in sample distribution and class range. To quantify the complexity of the classification tasks undertaken by different experts, a task complexity score is introduced as the basis for parameter lightweighting: Let the number of samples in the dataset subset processed by each expert be... The number of categories is The average number of samples per class is defined as Task complexity score The calculation formula is: ； In the formula, Represents the logarithmic function with base 2. , For the weighting coefficients, satisfying and The conditions are set to reflect the more dominant influence of the average number of samples in each class on the capacity of the neural network; With the task complexity score of the independent expert As a benchmark score : ； wherein , respectively represent the number of samples of the subset of the dataset handled by the independent expert, the number of classes. Let the task complexity score of the decoupling expert e be then the relative task complexity score ratio is: ； In the formula, e = 1, 2; the above ratio This reflects the ratio of the required neural network capacity of the decoupled expert e to that required by the independent expert; if This indicates that the decoupling expert e has low task complexity and can lightweight its network parameters; based on The number of channels of the last two residual convolution modules of the decoupling expert e is scaled, and the specific rules are as follows: ； wherein, is the number of channels decoupled from the expert e, is the number of channels for the independent expert, represents taking the nearest integer, and "adjust to even number ()" represents the rounding strategy, i.e., if the rounding still results in an odd number, the nearest even number is taken to ensure that the capacity is not excessively reduced.

6. The long-tail image recognition method based on hierarchical expert pair number of logarithmic probability fusion according to claim 5, characterized in that, To promote knowledge complementarity among experts and enhance the feature representation capabilities of shared-decoupled expert paths, the cross-expert feature fusion mechanism integrates the mid-level features of independent experts into the mid-level features of decoupled experts, as detailed below: The decoupled expert 1 feature fusion process is: Extracting the mid-level features of independent experts sequentially Mid-level features of decoupling expert 1 Align the channel count and width / height of the mid-layer features, then concatenate them. Next, perform convolutional fusion on the concatenated feature maps to obtain the fused features of Decoupling Expert 1. Finally, batch normalization and ReLU activation are performed to obtain the enhanced features of decoupling expert 1. , is represented as: ； ； The decoupled expert 2 feature fusion process is: Extracting mid-level features from independent experts sequentially Mid-layer features of Decoupling Expert 2 Align the channel count and width / height of the mid-layer features, then concatenate them. Next, perform convolutional fusion on the concatenated feature maps to obtain the fused features of Decoupling Expert 2. Finally, batch normalization and ReLU activation are performed to obtain the enhanced features of Decoupling Expert 2. , is represented as: ； ； wherein denotes a 1 x 1 convolution operation, denotes a feature alignment and concatenation operation, denotes a batch normalization operation.

7. The long-tail image recognition method based on hierarchical expert pair number probability fusion according to claim 6, characterized in that, Preliminary fusion is performed within the category intervals focused on by each expert, specifically by executing the following calculations: For the top category intervals, only independent experts are processed, and the log-odds output of the independent experts is used. For the middle category interval, both the independent expert and decoupling expert 1 are processed, and the log-probability outputs for both are provided. , Take the sum and average, that is For the tail category interval, output the logarithmic odds of independent expert, decoupling expert 1, and decoupling expert 2. , and Take the sum and average, that is ; The above calculations yielded preliminary fusion results.

8. The long-tail image recognition method based on hierarchical expert pair number of logarithmic probability fusion according to claim 7, characterized in that, A reconciliation correction term based on the output distribution difference is introduced to correct the preliminary fusion results, as follows: Probability distribution transformation and mean distribution calculation: Input the long-tailed image to be tested, obtain the log odds for each expert, first transform each expert... Log odds on category c Convert to probability distribution And calculate the average distribution. : ； In the formula, This represents the number of experts with log-probability outputs in category c, with values ​​of 1, 2, or 3. This represents the expert identifier, with values ​​1, 2, and 3, corresponding to independent expert, decoupling expert 1, and decoupling expert 2, respectively. Normalized distribution difference computation: Compute the degree of distribution difference between the expert output distribution and the average distribution based on the Jensen-Shannon divergence and normalize it: ； In the formula, represents the normalized distribution difference degree, represents a natural constant based logarithmic function, represents the calculation process of divergence, The greater, the greater the semantic output difference between experts; Harmonizing correction with final output: using the calculated degree of distribution difference, the preliminary fused logit is harmonized corrected to generate the final logit output : ； In the formula, To correct the intensity coefficient, the greater the semantic output discrepancy among experts, the less reliable the preliminary fusion result is. Therefore, by using a coordination correction term, the log probability result of the preliminary fusion is adaptively reduced, and then the probability distribution output is reduced after softmax normalization to lower the confidence for category c.

Images