Long-tail image recognition method based on neural network multi-expert hierarchical logic fusion

Abstract

The present invention discloses a long-tail image recognition method based on neural network multi-expert hierarchical logic fusion. The method is based on a neural network combined with multi-expert hierarchical logic fusion technology to achieve effective recognition of long-tail images. The method comprises a multi-branch feature extraction network, two multi-expert hierarchical logic fusion modules and a fully connected layer. The multi-branch feature extraction network comprises a shared backbone network and three independent expert networks. The two multi-expert hierarchical logic fusion modules are used to fuse the shallow logic output of the backbone network with the deep logic output of the first two expert networks. At the same time, a mutual learning method is used between the fused logic outputs of the expert networks, which not only promotes the three expert networks to transmit information and improve each other's performance, but also promotes the coordinated update of the backbone network parameters. The present invention can effectively enhance the network's recognition ability for minority class images in a long-tail image dataset and promote the coordinated update of the network's shallow and deep parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition and computer vision, and in particular to a long-tail image recognition method based on neural network multi-expert hierarchical logic fusion. Background Art

[0002] In the field of artificial intelligence, neural network technology has become a mainstream approach due to its outstanding feature extraction and pattern recognition capabilities, particularly in tasks such as image recognition, speech recognition, and natural language processing. However, in real-world applications, data often exhibits a long-tail distribution, where a minority class contains a large number of samples, while the majority class contains only a small number of samples. This data imbalance often leads to a decline in the recognition performance of standard neural network models, especially when classifying rare events or objects.

[0003] Existing technologies often use resampling, cost-sensitive learning, or traditional data balancing strategies to address the long-tail distribution problem. Category resampling balances the dataset by oversampling the minority category or undersampling the majority category, while cost-sensitive learning adjusts the classification loss function to assign a higher misclassification cost to the minority category. However, these methods often introduce additional noise or cause the model to over-focus on the minority category, thus affecting overall performance. Some research focuses on introducing attention mechanisms and feature fusion to enhance the model's ability to recognize infrequent categories. However, these methods fail to fully utilize the inherent hierarchical structure of the data and the correlation between categories. Sometimes, while increasing model complexity, they still fail to effectively improve the model's ability to handle long-tail data.

[0004] In addition, multi-model or ensemble learning methods propose improving recognition performance by combining the decisions of multiple classifiers. For example, bagging and boosting algorithms reduce the variance or bias of a model by integrating multiple learners. However, existing ensemble methods often fail to consider the logical relationships between individual classifiers. This is particularly true when processing long-tail data. They fail to effectively integrate the knowledge of multiple classifiers to compensate for data deficiencies, resulting in limited performance improvements in specific long-tail scenarios.

[0005] Recent advances have involved the design of expert systems, which are a collection of well-trained networks, each optimized for a specific subset of data. In long-tail recognition tasks, expert networks can be designed and trained specifically for the rare categories in the long-tail distribution. However, how to integrate the knowledge of these expert networks so that they work together remains a worthy challenge. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a long-tail image recognition method based on neural network multi-expert hierarchical logic fusion, which can effectively enhance the network's recognition ability for minority images in long-tail image datasets and promote the coordinated update of shallow and deep network parameters.

[0007] To achieve the above-mentioned purpose, the technical solution provided by the present invention is: a long-tail image recognition method based on multi-expert hierarchical logic fusion of a neural network, which realizes effective recognition of long-tail images based on a neural network combined with multi-expert hierarchical logic fusion technology. The neural network includes a multi-branch feature extraction network, two multi-expert hierarchical logic fusion modules and a fully connected layer as a classifier. The multi-branch feature extraction network includes a shared backbone network and three independent expert networks. The shared backbone network includes two residual convolution modules, and each expert network includes only one residual convolution module. The two multi-expert hierarchical logic fusion modules are used to fuse the shallow logic output of the shared backbone network with the deep logic output of the first two expert networks. The logic fusion can effectively enhance the network's recognition ability for long-tail image data sets. At the same time, a mutual learning method is used between the fused logic outputs of the expert networks. Mutual learning can not only promote the three expert networks to transmit information and improve each other's performance, but also promote the coordinated update of the backbone network parameters.

[0008] The specific implementation of the long-tail image recognition method based on neural network multi-expert hierarchical logic fusion includes:

[0009] The acquired long-tail image is input into the trained neural network that combines multi-expert hierarchical logic fusion technology to perform the following operations:

[0010] The long-tail image is first input into the multi-branch feature extraction network, which generates the first shallow feature map through the first residual convolution module of the shared backbone network. The first shallow feature map is then input into the second residual convolution module of the shared backbone network to extract the second shallow feature map. The second shallow feature map is then input into three expert networks in parallel. Each expert network generates a deep feature map through a residual convolution module.

[0011] After feature extraction is completed, the shallow logic output of the shared backbone network is fused with the deep logic output of the first two expert networks through the multi-expert hierarchical logic fusion module: first, all feature maps input to the multi-expert hierarchical logic fusion module are converted into logic outputs, the two shallow feature maps of the shared backbone network are converted into two shallow logic outputs through the fully connected layer, and the three deep feature maps of the three expert networks are converted into three deep logic outputs through the fully connected layer. Then, the first shallow logic output is fused with the deep logic output of the first expert network through the first multi-expert hierarchical logic fusion module to generate the fused logic output of the first expert network. The second shallow logic output is fused with the deep logic output of the second expert network through the second multi-expert hierarchical logic fusion module to generate the fused logic output of the second expert network. The fused logic output of the third expert network is the deep logic output of the third expert network.

[0012] After obtaining the fusion logic output of the three expert networks, the network's predictive reasoning is performed. In the predictive reasoning stage, the fusion logic output of the three expert networks is averaged to generate the final fusion logic output, which is the prediction result of the network's long-tail image category.

[0013] Furthermore, the generation process of the shallow feature map and the deep feature map is expressed as follows:

[0014] F s1 =θ1(x)

[0015] F s2 =θ2(θ1(x))

[0016] F1=θ 3_1 (θ2(θ1(x)))

[0017] F2=θ 3_2 (θ2(θ1(x)))

[0018] F3=θ 3_3 (θ2(θ1(x)))

[0019] Where θ1 and θ2 represent the parameter representation of the first and second residual convolution modules in the shared backbone network, respectively. 3_1 ,θ 3_2 and θ 3_3 Represent the parameter representation of the residual convolution module in the first, second and third expert networks respectively, x represents the input long-tail image, F s1 Represents the first shallow feature map of the shared backbone network, F s2 represents the second shallow feature map of the shared backbone network, and F1, F2, and F3 represent the deep feature maps generated by the first, second, and third expert networks, respectively.

[0020] Furthermore, the generation process of the fusion logic output of the three expert networks is expressed as:

[0021]

[0022] Where, and They represent the fused logical outputs of the three expert networks, α1 and α2 represent hyperparameters, and FC represents the fully connected layer, which is used to convert the feature map into the logical output.

[0023] Furthermore, during the training phase, the performance of the three expert networks is further improved by introducing mutual learning into the loss function;

[0024] The loss function L of the neural network is expressed as follows:

[0025]

[0026] Where x and y represent the input long-tail image and label respectively. and Represent the fusion logic outputs of the i-th and m-th expert networks respectively, D represents the training data set, M represents the number of experts set in the network, and KL represents the function for calculating the distillation loss between the two fusion logic outputs, which is used to achieve mutual learning;

[0027] In the prediction and inference stage, the fusion logic outputs of the three expert networks are averaged to generate the final fusion logic output, which is the prediction result of the network's long-tail image category;

[0028] The generation process of the final fusion logic output is expressed as:

[0029]

[0030] Where output represents the final fusion logic output, that is, the prediction result of the long-tail image category of the network.

[0031] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0032] 1. Enhanced network adaptability to different long-tail image data distributions. In traditional neural networks, a single network may be unable to recognize tail images in a long-tail image dataset due to the long-tail effect. Multi-expert networks can adaptively specialize networks for different image data subsets, thereby improving the overall prediction accuracy for tail images.

[0033] 2. The fusion strategy of shallow and deep logical outputs effectively solves the problem of information loss. In this invention, the detailed information captured by the low-level network is transmitted and integrated layer by layer, and can be optimized and adjusted at different levels of the network. This helps to extract deeper features in long-tail images, thereby improving the network's recognition ability.

[0034] 3. Mutual learning among multiple expert networks improves the overall generalization capability of the network. Each expert network not only learns from the training data but also calibrates and optimizes its own logical output by imitating the predictions of other expert networks. The information fusion between different expert networks enables each expert network to learn from different perspectives of other expert networks, further enhancing the overall generalization capability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a structural diagram of a neural network that combines multi-expert hierarchical logic fusion technology. In the figure, ResBlock represents the residual network module, Logit Fusion represents the multi-expert hierarchical logic fusion module, and Classifier represents the classifier (also a fully connected layer), which can convert feature maps into logical outputs. and Represent the fused logical outputs of the first, second, and third expert networks, respectively.

[0036] Figure 2 This is the structural diagram of the multi-expert hierarchical logic fusion module. In the figure, deep input represents the deep feature map of the expert network, shawllow input represents the shallow feature map of the shared backbone network, Classifier represents the classifier (also a fully connected layer), which can convert the feature map into a logic output, and α is the hyperparameter of the logic output fusion. It is the fusion logic output. DETAILED DESCRIPTION

[0037] The present invention will be further described below with reference to specific embodiments.

[0038] This embodiment discloses a long-tail image recognition method based on neural network multi-expert hierarchical logic fusion, which is based on a neural network combined with multi-expert hierarchical logic fusion technology to achieve effective recognition of long-tail images. Figure 1As shown, the neural network includes a multi-branch feature extraction network, two multi-expert hierarchical logic fusion modules and a fully connected layer as a classifier. The multi-branch feature extraction network includes a shared backbone network and three independent expert networks. The shared backbone network includes two residual convolution modules (ResBlock), which are responsible for preliminarily extracting the feature map of the long-tail image. Each expert network only contains one residual convolution module (ResBlock), which is responsible for further refining the extracted feature map. The two multi-expert hierarchical logic fusion modules are used to fuse the shallow logic output of the shared backbone network with the deep logic output of the first two expert networks. Logical fusion can effectively enhance the network's recognition ability for long-tail image datasets. At the same time, a mutual learning method is used between the fused logic outputs of the expert networks. Mutual learning can not only promote the three expert networks to transfer information and improve each other's performance, but also promote the coordinated update of the backbone network parameters.

[0039] First, we obtain a long-tail image dataset. This type of dataset is typically characterized by a rich head class with a limited number of images, while the tail class has a limited number of images, forming a typical long-tail distribution. This experiment uses the CIFAR-100-LT dataset, which includes versions with imbalance ratios of 50 and 100.

[0040] Next, we selected a deep learning network model for experimentation. We used ResNet32 as the core feature extraction network and established three expert networks to enhance the network's learning capabilities for tail-class images. The network was trained using the SGD optimizer with a momentum of 0.9 and a weight decay of 0.0005 to prevent overfitting. The base learning rate was set to 0.1.

[0041] After determining the long-tail dataset and the basic feature extraction network, the acquired long-tail images are input into the trained neural network that combines multi-expert hierarchical logic fusion technology to perform the following operations:

[0042] a. The long-tail image is first input into a multi-branch feature extraction network based on ResNet32. The first shallow feature map is generated by the first residual convolution module of the shared backbone network. The first shallow feature map is then input into the second residual convolution module of the shared backbone network to extract the second shallow feature map. The second shallow feature map is then input into three expert networks in parallel. Each expert network generates a deep feature map through a residual convolution module.

[0043] The generation process of the shallow feature map and the deep feature map is expressed as:

[0044] F s1 =θ1(x)

[0045] F s2=θ2(θ1(x))

[0046] F1=θ 3_1 (θ2(θ1(x)))

[0047] F2=θ 3_2 (θ2(θ1(x)))

[0048] F3=θ 3_3 (θ2(θ1(x)))

[0049] Where θ1 and θ2 represent the parameter representation of the first and second residual convolution modules in the shared backbone network, respectively. 3_1 ,θ 3_2 and θ 3_3 Represent the parameter representation of the residual convolution module in the first, second and third expert networks respectively, x represents the input long-tail image, F s1 Represents the first shallow feature map of the shared backbone network, F s2 represents the second shallow feature map of the shared backbone network, and F1, F2, and F3 represent the deep feature maps generated by the first, second, and third expert networks, respectively.

[0050] b. After feature extraction, the shallow logic output of the shared backbone network is fused with the deep logic output of the first two expert networks through the multi-expert hierarchical logic fusion module: First, all feature maps input to the multi-expert hierarchical logic fusion module are converted into logic outputs. The two shallow feature maps (shawllow input) of the shared backbone network are converted into two shallow logic outputs through the fully connected layer (Classfier). The three deep feature maps (deepinput) of the three expert networks are converted into three deep logic outputs through the fully connected layer (Classfier). Then, the first shallow logic output is fused with the deep logic output of the first expert network through the first multi-expert hierarchical logic fusion module (Logit Fusion) to generate the first

[0051] The fusion logic output of the expert network The second shallow logic output is fused with the deep logic output of the second expert network through the second multi-expert hierarchical logic fusion module (LogitFusion) to generate the fused logic output of the second expert network The fusion logic of the third expert network is The output is the deep logic output of the third expert network;

[0052] The generation process of the fusion logic output of the three expert networks is expressed as:

[0053]

[0054] Where, and They represent the fused logical outputs of the three expert networks, α1 and α2 represent hyperparameters, and FC represents the fully connected layer, which is used to convert the feature map into the logical output.

[0055] c. After obtaining the fused logical outputs of the three expert networks, the network is used for prediction and inference. During the prediction and inference phase, the fused logical outputs of the three expert networks are averaged to generate the final fused logical output, which is the prediction result of the network's long-tail image category.

[0056] The generation process of the final fusion logic output is expressed as:

[0057]

[0058] Where output represents the final fusion logic output, that is, the prediction result of the long-tail image category of the network.

[0059] Specifically, during the training phase of the neural network, the performance of the three expert networks is further improved by introducing mutual learning into the loss function. The loss function L of the neural network is expressed as follows:

[0060]

[0061] Where x and y represent the input long-tail image and label respectively. and They represent the fused logic outputs of the i-th and m-th expert networks respectively, D represents the training dataset, M represents the number of experts set in the network, and KL represents the function for calculating the distillation loss between the two fused logic outputs, which is used to achieve mutual learning.

[0062] In order to verify the superiority of the method of the present invention, the present invention is tested on the CIFAR-100-LT dataset. The test results are shown in Table 1.

[0063] Table 1 Image classification accuracy of each method on the CIFAR-100-LT dataset (%)

[0064]

[0065] Table 1 compares the accuracy of various long-tail learning methods on the CIFAR-100-LT dataset. The baseline is the ResNet32 basic image recognition model. LDAM_DRW and Balanced Softmax both balance the long-tail image recognition problem from the perspective of the loss function. RIDE and SADE also use multi-expert networks to solve the long-tail image recognition problem. As shown in the figure, compared to the previously most accurate SADE method, the accuracy of this method is 0.3% higher on the CIFAR-100-LT dataset with an imbalance ratio of 100, and 0.5% higher on the CIFAR-100-LT dataset with an imbalance ratio of 50.

[0066] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, any changes made based on the shape and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A long-tail image recognition method based on neural network multi-expert hierarchical logic fusion, characterized by: This method is based on a neural network that combines multi-expert hierarchical logic fusion technology to achieve effective recognition of long-tail images. The neural network includes a multi-branch feature extraction network, two multi-expert hierarchical logic fusion modules and a fully connected layer as a classifier. The multi-branch feature extraction network includes a shared backbone network and three independent expert networks. The shared backbone network includes two residual convolution modules, and each expert network only includes one residual convolution module. The two multi-expert hierarchical logic fusion modules are used to fuse the shallow logic output of the shared backbone network with the deep logic output of the first two expert networks. Logical fusion can effectively enhance the network's recognition ability for long-tail image datasets. At the same time, a mutual learning method is used between the fused logic outputs of the expert networks. Mutual learning not only promotes the information transmission among the three expert networks and improves each other's performance, but also promotes the coordinated update of the backbone network parameters. The specific implementation of the long-tail image recognition method based on neural network multi-expert hierarchical logic fusion includes: The acquired long-tail image is input into the trained neural network that combines multi-expert hierarchical logic fusion technology to perform the following operations: The long-tail image is first input into the multi-branch feature extraction network, which generates the first shallow feature map through the first residual convolution module of the shared backbone network. The first shallow feature map is then input into the second residual convolution module of the shared backbone network to extract the second shallow feature map. The second shallow feature map is then input into three expert networks in parallel. Each expert network generates a deep feature map through a residual convolution module. After feature extraction is completed, the shallow logic output of the shared backbone network is fused with the deep logic output of the first two expert networks through the multi-expert hierarchical logic fusion module: first, all feature maps input to the multi-expert hierarchical logic fusion module are converted into logic outputs, the two shallow feature maps of the shared backbone network are converted into two shallow logic outputs through the fully connected layer, and the three deep feature maps of the three expert networks are converted into three deep logic outputs through the fully connected layer. Then, the first shallow logic output is fused with the deep logic output of the first expert network through the first multi-expert hierarchical logic fusion module to generate the fused logic output of the first expert network. The second shallow logic output is fused with the deep logic output of the second expert network through the second multi-expert hierarchical logic fusion module to generate the fused logic output of the second expert network. The fused logic output of the third expert network is the deep logic output of the third expert network. After obtaining the fusion logic output of the three expert networks, the network's predictive reasoning is performed. In the predictive reasoning stage, the fusion logic output of the three expert networks is averaged to generate the final fusion logic output, which is the prediction result of the network's long-tail image category.

2. The long-tail image recognition method based on neural network multi-expert hierarchical logic fusion according to claim 1 is characterized in that: The generation process of the shallow feature map and the deep feature map is expressed as: F s1 =θ1(x) F s2 =θ2(θ1(x)) F1=θ 3_1 (θ2(θ1(x))) F2=θ 3_2 (θ2(θ1(x))) F3=θ 3_3 (θ2(θ1(x))) Where θ1 and θ2 represent the parameter representation of the first and second residual convolution modules in the shared backbone network, respectively. 3_1 ,θ 3_2 and θ 3_3 Represent the parameter representation of the residual convolution module in the first, second and third expert networks respectively, x represents the input long-tail image, F s1 Represents the first shallow feature map of the shared backbone network, F s2 represents the second shallow feature map of the shared backbone network, and F1, F2, and F3 represent the deep feature maps generated by the first, second, and third expert networks, respectively.

3. The long-tail image recognition method based on neural network multi-expert hierarchical logic fusion according to claim 2 is characterized in that: The generation process of the fusion logic output of the three expert networks is expressed as: Where, and They represent the fused logical outputs of the three expert networks, α1 and α2 represent hyperparameters, and FC represents the fully connected layer, which is used to convert the feature map into the logical output.

4. The long-tail image recognition method based on neural network multi-expert hierarchical logic fusion according to claim 3 is characterized in that: During the training phase, the performance of the three expert networks is further improved by introducing mutual learning into the loss function; The loss function L of the neural network is expressed as follows: Where x and y represent the input long-tail image and label respectively. and Represent the fusion logic outputs of the i-th and m-th expert networks respectively, D represents the training data set, M represents the number of experts set in the network, and KL represents the function for calculating the distillation loss between the two fusion logic outputs, which is used to achieve mutual learning; In the prediction and inference stage, the fusion logic outputs of the three expert networks are averaged to generate the final fusion logic output, which is the prediction result of the network's long-tail image category; The generation process of the final fusion logic output is expressed as: Where output represents the final fusion logic output, that is, the prediction result of the long-tail image category of the network.

Images