Long-tail image recognition method based on deep learning branch feature map fusion

Abstract

The present invention discloses a long-tail image recognition method based on deep learning branch feature map fusion, the method is to realize effective recognition of long-tail images based on deep learning branch feature map fusion network, the deep learning branch feature map fusion network includes a multi-branch feature extraction network, two branch feature map fusion modules and three fully connected layers as classifiers, the multi-branch feature extraction network includes a shared network and three branch networks, the shared network includes a residual convolution module, each branch network includes two residual convolution modules, the branch feature map fusion module is used to fuse the deep feature map of the branch network with the shallow feature map of other branch networks, and at the same time use a mutual learning method between the logical outputs of the branch networks. The present invention can improve the network's classification ability for long-tail images by fusing deep feature maps with shallow feature maps, and at the same time improve the overall performance of the deep learning network by mutual learning.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition and computer vision, and in particular to a long-tail image recognition method based on deep learning branch feature map fusion. Background Art

[0002] The long-tail distribution problem is a major challenge in machine learning and computer vision. In many practical applications, such as image recognition and natural language processing, datasets often exhibit long-tail distribution characteristics, meaning that the dataset has a large number of categories but the number of samples in each category is extremely unbalanced. This imbalance causes the model to perform well in identifying the head categories (i.e., categories with a large number of samples), but perform poorly on the tail categories (i.e., categories with a small number of samples).

[0003] Existing recognition methods often focus on enhancing the model's performance on head categories and ignore the recognition accuracy of tail categories. Since the model encounters head category data more frequently during training, it can better learn the characteristics of these categories. However, due to the scarcity of samples in tail categories, it is difficult for the model to learn effective distinguishing features from them. This directly affects the model's adaptability to tail categories, thereby affecting the model's overall effectiveness and wide application.

[0004] Traditional machine learning methods are prone to performance degradation when training long-tail datasets. Due to the small number of samples in the minority category, the classifier cannot fully learn the characteristics and patterns of these categories, resulting in poor discrimination ability. At the same time, the majority class has a large amount of data used to train the classifier and is given a higher weight, while the minority class data is given a lower weight. If the classifier does not consider the importance of the minority class data, it will lead to performance degradation. This unbalanced distribution ultimately leads to the classifier being more inclined to classify samples into the more frequent category and ignore the minority class. This often results in poor classification of the minority class, which limits the accuracy and generalization ability of the classifier.

[0005] To address this problem, scholars have proposed a variety of methods. One common strategy is resampling technology, which alleviates category imbalance by artificially adjusting the sample size of each category. However, this method may introduce the risk of overfitting and cannot fundamentally improve the recognition ability of tail categories. Another strategy is cost-sensitive learning, which strengthens the model's learning of these categories by increasing the loss function of tail categories. Although these methods improve the recognition rate of tail categories to a certain extent, they still face the challenges of high computational complexity and difficulty adapting to new categories. In recent years, knowledge distillation and learning transfer strategies have been proposed to deal with long-tail recognition problems. These methods improve performance by transferring the knowledge of a pre-trained large model to a small model optimized for a specific task. However, these methods usually rely on a single source of knowledge and lack effective utilization of multi-source information. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a long-tail image recognition method based on the fusion of deep learning branch feature maps. The method can improve the network's classification ability for long-tail images by fusing deep feature maps with shallow feature maps, and at the same time improve the overall performance of the deep learning network through mutual learning.

[0007] To achieve the above-mentioned purpose, the technical solution provided by the present invention is: a long-tail image recognition method based on deep learning branch feature map fusion, which realizes effective recognition of long-tail images based on a deep learning branch feature map fusion network. The deep learning branch feature map fusion network includes a multi-branch feature extraction network, two branch feature map fusion modules and three fully connected layers as classifiers. The multi-branch feature extraction network includes a shared network and three branch networks. The shared network includes a residual convolution module, and each branch network includes two residual convolution modules. The branch feature map fusion module is used to fuse the deep feature map of the branch network with the shallow feature map of other branch networks to improve the network's attention to the tail class image, and the fusion of feature maps of different branch networks can promote the diversified output of network feature maps. At the same time, a mutual learning method is used between the logical outputs of the branch networks to ensure that each branch network can further learn and adjust parameters from the data of other branch networks after the branch feature maps are fused;

[0008] The specific implementation of the long-tail image recognition method based on deep learning branch feature map fusion includes:

[0009] Input the acquired long-tail image into the trained deep learning branch feature map fusion network and perform the following operations:

[0010] The long-tail image is first input into the multi-branch feature extraction network. The shared network extracts features from the input long-tail image to generate a primary feature map. The primary feature map is then input into the three subsequent branch networks. Each branch network continues to extract features. The primary feature map is passed through the first residual convolution module of the branch network to generate a shallow feature map of the branch network. The shallow feature map is passed through the second residual convolution module of the branch network to generate a deep feature map of the branch network.

[0011] After feature extraction is completed, the feature maps of the three branch networks are fused through two branch feature map fusion modules. The process is as follows: the deep feature map of the first branch network is fused with the shallow feature map of the second branch network through the first branch feature map fusion module, and the deep feature map of the second branch network is fused with the shallow feature map of the third branch network through the second branch feature map fusion module, while the feature map of the third branch network remains unchanged;

[0012] The fused feature map output by the first branch feature map fusion module is input into the first fully connected layer to obtain the logical output of the first branch network. The fused feature map output by the second branch feature map fusion module is input into the second fully connected layer to obtain the logical output of the second branch network. The deep feature map of the third branch network is input into the third fully connected layer to obtain the logical output of the third branch network. The logical outputs of each branch network are averaged to generate the final logical output, which is the prediction result of the long-tail image category of the network.

[0013] Furthermore, the shallow and deep feature map generation process of the first branch network is expressed as:

[0014] F 2_1 =θ 2_1 (θ1(x))

[0015] F 3_1 =θ 3_1 (θ 2_1 (θ1(x)))

[0016] Where, F 2_1 、F 3_1 Represent the shallow and deep feature maps of the first branch network, θ 2_1 ,θ 3_1 They represent the parameter representations of the first and second residual convolution modules of the first branch network, θ1 is the parameter representation of the residual convolution module of the shared network, and x represents the input long-tail image;

[0017] The generation process of the shallow and deep feature maps of the second branch network is expressed as:

[0018] F 2_2 =θ 2_2(θ1(x))

[0019] F 3_2 =θ 3_2 (θ 2_2 (θ1(x)))

[0020] Where, F 2_2 、F 3_2 Represent the shallow and deep feature maps of the second branch network, θ 2_2 ,θ 3_2 Represent the parameter representation of the first and second residual convolution modules of the second branch network respectively;

[0021] The generation process of the shallow and deep feature maps of the third branch network is expressed as:

[0022] F 2_3 =θ 2_3 (θ1(x))

[0023] F 3_3 =θ 3_3 (θ 2_3 (θ1(x)))

[0024] Where, F 2_3 、F 3_3 Represent the shallow and deep feature maps of the third branch network, θ 2_3 ,θ 3_3 Represent the parameter representations of the first and second residual convolution modules of the third branch network respectively.

[0025] Furthermore, after obtaining the shallow and deep feature maps of the branch network, the feature maps are fused through two branch feature map fusion modules, as follows:

[0026] The deep feature map of the first branch network is fused with the shallow feature map of the second branch network through the first branch feature map fusion module. The specific operations of the fusion process are as follows:

[0027] Use strided convolution to downsample the shallow feature map of the second branch network, and align the number of channels and length and width of the deep feature map of the first branch network with the shallow feature map of the second branch network. Then, splice the two feature maps, perform convolution on the spliced ​​feature map, adjust the number of channels and length and width of the spliced ​​feature map, and finally output the fused feature map of the first branch network.

[0028] The feature map fusion process of the first branch network is expressed as:

[0029] F F1 =concat[F 3_1 ,conv stride＝2 (F2_2 )]

[0030] F O1 =BN(conv 1*1 (BN(conv 3*3 (F F1 ))))

[0031] Where, F F1 Represents the feature map after the deep feature map of the first branch network and the shallow feature map of the second branch network are spliced ​​together, F O1 Represents the fusion feature map of the first branch network, conv stride＝2 represents a convolution operation with a stride of 2, concat represents a feature map concatenation operation, conv 3*3 、conv 1*1 They represent convolution operations using 3*3 and 1*1 convolution kernels respectively, and BN represents batch normalization operations;

[0032] The deep feature map of the second branch network is fused with the shallow feature map of the third branch network. The specific operations of the fusion process are as follows:

[0033] Use strided convolution to downsample the shallow feature map of the third branch network, and align the number of channels and length and width of the deep feature map of the second branch network with the shallow feature map of the third branch network. Then, splice the two feature maps, perform convolution on the spliced ​​feature map, adjust the number of channels and length and width of the spliced ​​feature map, and finally output the fused feature map of the second branch network.

[0034] The feature map fusion process of the second branch network is expressed as:

[0035] F F2 =concat[F 3_2 ,conv stride＝2 (F 2_3 )]

[0036] F O2 =BN(conv 1*1 (BN(conv 3*3 (F F2 ))))

[0037] Where, F F1 It represents the feature map after the deep feature map of the second branch network and the shallow feature map of the third branch network are spliced ​​together, F O2 Represents the fused feature map of the second branch network;

[0038] The feature map of the third branch network remains unchanged, and the deep feature map of the third branch network is directly output. The output process is expressed as:

[0039] F O3 =θ 3_3 (θ 2_3 (θ1(x)))

[0040] Where, F O3 Represents the deep feature map of the third branch network.

[0041] Furthermore, the feature map F O1 、F O2 and F O3 The corresponding fully connected layers are input respectively to obtain the logical outputs P1, P2 and P3 of each branch. During network training, the logical outputs of each branch network are adjusted by introducing mutual distillation loss to adjust the overall loss function of the network, thereby achieving mutual learning between the branch networks. The loss function formula of the deep learning branch feature map fusion network is as follows:

[0042]

[0043] L=L CE +λL KL

[0044] Where, P s 、P t Represent the logical outputs of the sth and tth branch networks, P s c 、P t c Respectively represent the logical outputs of the sth and tth branch networks on the cth category, τ and λ are hyperparameters, C represents the total number of categories, M represents the number of branches, KL(P s ||P t ) is used to calculate the distillation loss of the distribution gap between two logical outputs, L KL Represents the sum of the distillation losses generated by the logical outputs of all branch networks, L CE represents the cross entropy loss, L represents the overall loss of the network, and y c Indicates the label of the input data in the cth category;

[0045] During network prediction and inference, the logical outputs of each branch network are averaged to generate the final logical output, which is the prediction result of the network's long-tail image category.

[0046] The generation process of the final logical output is expressed as:

[0047] output=(P1+P2+P3) / 3

[0048] Where output represents the final logical output of the network, that is, the prediction result of the long-tail image category of the network.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] 1. Improve the recognition accuracy of tail-like images in long-tail images. Deep features have good performance in recognizing head-like images in long-tail images, while shallow features are more effective in recognizing tail-like images in many long-tail images. To this end, fusing the deep feature maps of a branch network with the shallow feature maps of other branch networks can effectively alleviate the problem of long-tail images. At the same time, fusing the feature maps of different branch networks can also promote the diversification of network feature maps.

[0051] 2. Reduce the risk of network overfitting. In the branch network, each branch network independently learns from the primary feature map of the shared network, which helps to reduce the risk of overfitting of a single branch network. At the same time, the mutual learning strategy further ensures that each branch network is not limited to its own learning area, but can learn and adjust from the information of other branch networks.

[0052] 3. Enhance the generalization ability of the network. By fusing feature map information from different branch networks, the network can obtain a more comprehensive data representation, thereby enhancing its generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a structural diagram of the deep learning branch feature map fusion network; in the figure, ResBlock represents the residual convolution module, Fusion represents the branch feature map fusion module, Classfier represents the classifier (also a fully connected layer), which can convert the feature map into logical output, and Logit1, Logit2 and Logit3 represent the logical outputs of the first, second and third branch networks respectively.

[0054] Figure 2 This is the structural diagram of the branch feature map fusion module; in the figure, deep input represents the deep feature map of the branch network, shawllow input represents the shallow feature map of other branch networks, DownSampling represents the downsampling operation, c represents the feature map splicing operation, and Conv Layers represents the convolution layer. DETAILED DESCRIPTION

[0055] The present invention will be further described below with reference to specific embodiments.

[0056] This embodiment discloses a long-tail image recognition method based on deep learning branch feature map fusion, which is based on a deep learning branch feature map fusion network to achieve effective recognition of long-tail images. Figure 1As shown, the deep learning branch feature map fusion network includes a multi-branch feature extraction network, two branch feature map fusion modules (Fusion) and three fully connected layers (Classfier) ​​as classifiers. The multi-branch feature extraction network includes a shared network and three branch networks. The shared network includes a residual convolution module (ResBlock). Each branch network includes two residual convolution modules (ResBlock). The residual convolution module (ResBlock) is a common structure in deep learning. It aims to solve the gradient disappearance problem caused by the network being too deep. It consists of a series of convolution layers, activation layers and batch normalization layers. The branch feature map fusion module is used to fuse the deep feature map of the branch network with the shallow feature map of other branch networks to improve the network's attention to the tail class image, and the fusion of feature maps of different branch networks can promote the diversified output of network feature maps. At the same time, a mutual learning method is used between the logical outputs of the branch networks to ensure that each branch network can further learn and adjust parameters from the data of other branch networks after the branch feature map is fused. The specific implementation of this method includes:

[0057] First, we obtain a long-tail image dataset. Common long-tail image datasets include the iNaturalist dataset, the ImageNet-LT dataset, and the CIFAR10-LT and CIFAR100-LT datasets. These image datasets are characterized by having a large number of images in the head category, while most categories in the tail have only a very small number of images, resulting in a significant long-tail distribution. The present invention uses the CIFAR100-LT dataset, using image datasets with imbalance ratios of 50 and 100.

[0058] The feature extraction of the long-tail image dataset requires the use of a deep learning network. Therefore, ResNet32 is used as the basic feature extraction network with 3 branch networks. The SGD optimizer is used. In terms of parameter settings, the momentum is set to 0.9, the weight decay is set to 0.0005, and the basic learning rate is 0.1.

[0059] After determining the long-tail image dataset and the basic feature extraction network, the acquired long-tail images are input into the trained deep learning branch feature map fusion network to perform the following operations:

[0060] a. The long-tail image is first input into the multi-branch feature extraction network based on ResNet32. The shared network extracts features from the input long-tail image and generates a primary feature map. The primary feature map is then input into the three subsequent branch networks. Each branch network continues to extract features. The primary feature map is passed through the first residual convolution module of the branch network to generate the shallow feature map of the branch network. The shallow feature map is passed through the second residual convolution module of the branch network to generate the deep feature map of the branch network.

[0061] The shallow and deep feature map generation process of the first branch network is expressed as:

[0062] F 2_1 =θ 2_1 (θ1(x))

[0063] F 3_1 =θ 3_1 (θ 2_1 (θ1(x)))

[0064] Where, F 2_1 、F 3_1 Represent the shallow and deep feature maps of the first branch network, θ 2_1 ,θ 3_1 They represent the parameter representations of the first and second residual convolution modules of the first branch network, θ1 is the parameter representation of the residual convolution module of the shared network, and x represents the input long-tail image;

[0065] The generation process of the shallow and deep feature maps of the second branch network is expressed as:

[0066] F 2_2 =θ 2_2 (θ1(x))

[0067] F 3_2 =θ 3_2 (θ 2_2 (θ1(x)))

[0068] Where, F 2_2 、F 3_2 Represent the shallow and deep feature maps of the second branch network, θ 2_2 ,θ 3_2 Represent the parameter representation of the first and second residual convolution modules of the second branch network respectively;

[0069] The generation process of the shallow and deep feature maps of the third branch network is expressed as:

[0070] F 2_3 =θ 2_3 (θ1(x))

[0071] F 3_3 =θ 3_3 (θ 2_3 (θ1(x)))

[0072] Where, F 2_3 、F 3_3 Represent the shallow and deep feature maps of the third branch network, θ 2_3 ,θ 3_3Represent the parameter representations of the first and second residual convolution modules of the third branch network respectively.

[0073] b. After feature extraction, the feature maps of the three branch networks are fused through two branch feature map fusion modules, such as Figure 2 As shown, the details are as follows:

[0074] The deep feature map of the first branch network is fused with the shallow feature map of the second branch network through the first branch feature map fusion module. The specific operations of the fusion process are as follows:

[0075] Use strided convolution to downsample the shallow feature map (shawllow input) and align the number of channels and length and width of the deep feature map (deep input) of the first branch network with the shallow feature map (shawllowinput) of the second branch network. Then, splice the two feature maps, perform convolution on the spliced ​​feature map, adjust the number of channels and length and width of the spliced ​​feature map, and finally output the fused feature map of the first branch network;

[0076] The feature map fusion process of the first branch network is expressed as:

[0077] F F1 =concat[F 3_1 ,conv stride＝2 (F 2_2 )]

[0078] F O1 =BN(conv 1*1 (BN(conv 3*3 (F F1 ))))

[0079] Where, F F1 Represents the feature map after the deep feature map of the first branch network is concatenated with the shallow feature map of the second branch network, F O1 Represents the fusion feature map of the first branch network, conv stride＝2 represents a convolution operation with a stride of 2, concat represents a feature map concatenation operation, conv 3*3 、conv 1*1 They represent convolution operations using 3*3 and 1*1 convolution kernels respectively, and BN represents batch normalization operations;

[0080] The deep feature map of the second branch network is fused with the shallow feature map of the third branch network. The specific operations of the fusion process are as follows:

[0081] Use strided convolution to downsample the shallow feature map (shawllow input) and align the number of channels and length and width of the deep feature map (deep input) of the second branch network with the shallow feature map (shawllowinput) of the third branch network. Then, splice the two feature maps, perform convolution on the spliced ​​feature map, adjust the number of channels and length and width of the spliced ​​feature map, and finally output the fused feature map of the second branch network;

[0082] The feature map fusion process of the second branch network is expressed as:

[0083] F F2 =concat[F 3_2 ,conv stride＝2 (F 2_3 )]

[0084] F O2 =BN(conv 1*1 (BN(conv 3*3 (F F2 ))))

[0085] Where, F F1 It represents the feature map after the deep feature map of the second branch network and the shallow feature map of the third branch network are spliced ​​together, F O2 Represents the fused feature map of the second branch network;

[0086] The feature map of the third branch network remains unchanged, and the deep feature map of the third branch network is directly output. The output process is expressed as:

[0087] F O3 =θ 3_3 (θ 2_3 (θ1(x)))

[0088] Where, F O3 Represents the deep feature map of the third branch network.

[0089] c. The feature map F O1 、F O2 and F O3 The corresponding fully connected layers are input respectively to obtain the logical outputs P1, P2 and P3 of each branch. During network training, the logical outputs of each branch network are adjusted by introducing mutual distillation loss to adjust the overall loss function of the network, thereby achieving mutual learning between the branch networks. The loss function formula of the deep learning branch feature map fusion network is as follows:

[0090]

[0091] L=LCE +λL KL

[0092] Where, P s 、P t Represent the logical outputs of the sth and tth branch networks, P s c 、P t c Respectively represent the logical outputs of the sth and tth branch networks on the cth category, τ and λ are hyperparameters, C represents the total number of categories, M represents the number of branches, KL(P s ||P t ) is used to calculate the distillation loss of the distribution gap between two logical outputs, L KL Represents the sum of the distillation losses generated by the logical outputs of all branch networks, L CE represents the cross entropy loss, L represents the overall loss of the network, and y c Indicates the label of the input data in the cth category;

[0093] During network prediction and inference, the logical outputs of each branch network are averaged to generate the final logical output, which is the prediction result of the network's long-tail image category.

[0094] The generation process of the final logical output is expressed as:

[0095] output=(P1+P2+P3) / 3

[0096] Where output represents the final logical output of the network, that is, the prediction result of the long-tail image category of the network.

[0097] In order to verify the superiority of the present invention, experiments are conducted on the CIFAR100-LT dataset. The experimental results are shown in Table 1.

[0098] Table 1 Image classification accuracy of each method on the CIFAR100-LT dataset (%)

[0099]

[0100] Table 1 compares the image prediction accuracy of various methods on the CIFAR-100-LT dataset. Baseline is the ResNet32 basic image recognition model. LDAM_DRW and Balanced Softmax both balance the long-tailed image recognition problem from the perspective of the loss function. BBN uses a dual-branch structure to process head and tail image data respectively. LADE, RIDE, and SADE also use multi-expert networks to solve the long-tailed image recognition problem. As can be seen from the data in the figure, compared to the previously most accurate SADE method, the accuracy of this method is improved by 1.8% on the CIFAR-100-LT dataset with an imbalance ratio of 100, and by 1.2% on the CIFAR-100-LT dataset with an imbalance ratio of 50.

[0101] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, any changes made based on the shape and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A long-tail image recognition method based on deep learning branch feature map fusion, characterized by: This method is based on a deep learning branch feature map fusion network to achieve effective recognition of long-tail images. The deep learning branch feature map fusion network includes a multi-branch feature extraction network, two branch feature map fusion modules and three fully connected layers as classifiers. The multi-branch feature extraction network includes a shared network and three branch networks. The shared network includes a residual convolution module, and each branch network includes two residual convolution modules. The branch feature map fusion module is used to fuse the deep feature map of the branch network with the shallow feature map of other branch networks to improve the network's attention to tail-class images. The fusion of feature maps of different branch networks can promote the diversified output of network feature maps. At the same time, a mutual learning method is used between the logical outputs of the branch networks to ensure that each branch network can further learn and adjust parameters from the data of other branch networks after the branch feature maps are fused. The specific implementation of the long-tail image recognition method based on deep learning branch feature map fusion includes: Input the acquired long-tail image into the trained deep learning branch feature map fusion network and perform the following operations: The long-tail image is first input into the multi-branch feature extraction network. The shared network extracts features from the input long-tail image to generate a primary feature map. The primary feature map is then input into the three subsequent branch networks. Each branch network continues to extract features. The primary feature map is passed through the first residual convolution module of the branch network to generate a shallow feature map of the branch network. The shallow feature map is passed through the second residual convolution module of the branch network to generate a deep feature map of the branch network. After feature extraction is completed, the feature maps of the three branch networks are fused through two branch feature map fusion modules. The process is as follows: the deep feature map of the first branch network is fused with the shallow feature map of the second branch network through the first branch feature map fusion module, and the deep feature map of the second branch network is fused with the shallow feature map of the third branch network through the second branch feature map fusion module, while the feature map of the third branch network remains unchanged; The fused feature map output by the first branch feature map fusion module is input into the first fully connected layer to obtain the logical output of the first branch network. The fused feature map output by the second branch feature map fusion module is input into the second fully connected layer to obtain the logical output of the second branch network. The deep feature map of the third branch network is input into the third fully connected layer to obtain the logical output of the third branch network. The logical outputs of each branch network are averaged to generate the final logical output, which is the prediction result of the long-tail image category of the network.

2. The long-tail image recognition method based on deep learning branch feature map fusion according to claim 1 is characterized in that: The shallow feature map and deep feature map generation process of the first branch network are expressed as: F 2_1 =θ 2_1 (θ1(x)) F 3_1 =θ 3_1 (i 2_1 (θ1(x))) Where, F 2_1 、F 3_1 Represent the shallow feature map and deep feature map of the first branch network respectively, θ 2_1 ,θ 3_1 They represent the parameter representations of the first and second residual convolution modules of the first branch network, θ1 is the parameter representation of the residual convolution module of the shared network, and x represents the input long-tail image; The generation process of the shallow feature map and the deep feature map of the second branch network is expressed as: F 2_2 =θ 2_2 (θ1(x)) F 3_2 =θ 3_2 (i 2_2 (θ1(x))) Where, F 2_2 、F 3_2 Represent the shallow feature map and deep feature map of the second branch network respectively, θ 2_2 ,θ 3_2 Represent the parameter representation of the first and second residual convolution modules of the second branch network respectively; The generation process of the shallow feature map and the deep feature map of the third branch network is expressed as: F 2_3 =θ 2_3 (θ1(x)) F 3_3 =θ 3_3 (i 2_3 (θ1(x))) Where, F 2_3 、F 3_3 Represent the shallow feature map and deep feature map of the third branch network respectively, θ 2_3 ,θ 3_3 Represent the parameter representations of the first and second residual convolution modules of the third branch network respectively.

3. The long-tail image recognition method based on deep learning branch feature map fusion according to claim 2 is characterized in that: After obtaining the shallow feature map and deep feature map of the branch network, the feature maps are fused through two branch feature map fusion modules as follows: The deep feature map of the first branch network is fused with the shallow feature map of the second branch network through the first branch feature map fusion module. The specific operations of the fusion process are as follows: Use strided convolution to downsample the shallow feature map of the second branch network, and align the number of channels and length and width of the deep feature map of the first branch network with the shallow feature map of the second branch network. Then, splice the two feature maps, perform convolution on the spliced ​​feature map, adjust the number of channels and length and width of the spliced ​​feature map, and finally output the fused feature map of the first branch network. The feature map fusion process of the first branch network is expressed as: F F1 =concat[F 3_1 ,conv stride＝2 (F 2_2 )] F O1 =BN(conv 1*1 (BN(conv 3*3 (F F1 )))) Where, F F1 Represents the feature map after the deep feature map of the first branch network and the shallow feature map of the second branch network are spliced ​​together, F O1 Represents the fusion feature map of the first branch network, conv stride＝2 represents a convolution operation with a stride of 2, concat represents a feature map concatenation operation, conv 3*3 、conv 1*1 They represent convolution operations using 3*3 and 1*1 convolution kernels respectively, and BN represents batch normalization operations; The deep feature map of the second branch network is fused with the shallow feature map of the third branch network. The specific operations of the fusion process are as follows: Use strided convolution to downsample the shallow feature map of the third branch network, and align the number of channels and length and width of the deep feature map of the second branch network with the shallow feature map of the third branch network. Then, splice the two feature maps, perform convolution on the spliced ​​feature map, adjust the number of channels and length and width of the spliced ​​feature map, and finally output the fused feature map of the second branch network. The feature map fusion process of the second branch network is expressed as: F F2 =concat[F 3_2 ,conv stride＝2 (F 2_3 )] F O2 =BN(conv 1*1 (BN(conv 3*3 (F F2 )))) Where, F F2 It represents the feature map after the deep feature map of the second branch network and the shallow feature map of the third branch network are spliced ​​together, F O2 Represents the fused feature map of the second branch network; The feature map of the third branch network remains unchanged, and the deep feature map of the third branch network is directly output. The output process is expressed as: F O3 =θ 3_3 (i 2_3 (θ1(x))) Where, F O3 Represents the deep feature map of the third branch network.

4. The long-tail image recognition method based on deep learning branch feature map fusion according to claim 3 is characterized in that: The feature map F O1 、F O2 and F O3 The corresponding fully connected layers are input respectively to obtain the logical outputs P1, P2 and P3 of each branch. During network training, the logical outputs of each branch network are adjusted by introducing mutual distillation loss to adjust the overall loss function of the network, thereby achieving mutual learning between the branch networks. The loss function formula of the deep learning branch feature map fusion network is as follows: L=L CE +λL KL Where, P s 、P t Represent the logical outputs of the sth and tth branch networks, P s c 、P t c Respectively represent the logical outputs of the sth and tth branch networks on the cth category, τ and λ are hyperparameters, C represents the total number of categories, M represents the number of branches, KL(P s ||P t ) is used to calculate the distillation loss of the distribution gap between two logical outputs, L KL Represents the sum of the distillation losses generated by the logical outputs of all branch networks, L CE represents the cross entropy loss, L represents the overall loss of the network, and y c Indicates the label of the input data in the cth category; During network prediction and inference, the logical outputs of each branch network are averaged to generate the final logical output, which is the prediction result of the network's long-tail image category. The generation process of the final logical output is expressed as: output=(P1+P2+P3) / 3 Where output represents the final logical output of the network, that is, the prediction result of the long-tail image category of the network.

Images