Multi-branch selection fusion fine-grained classification method based on coordinate attention positioning

Abstract

The invention provides a multi-branch selection fusion fine-grained classification method based on coordinate attention positioning, which relates to the field of image processing, and comprises the following steps: establishing a multi-branch selection fusion fine-grained classification model, the selection branch distribution space pool pyramid and region coordinate attention object positioning module is used for generating a target image according to the overall features of the original image, the object branches are used for extracting the features of the target image, and the first selection branch block is used for fusing the overall features of the original image and the features of the target image to generate fused features; the attention part suggestion module is used for cutting the target image into a plurality of part images according to the fused features, and the part branch is used for generating a classification result according to the plurality of part images; and classifying the to-be-classified image through the trained multi-branch selection fusion fine-grained classification model. The method has the advantage of improving the efficiency and accuracy of fine-grained visual classification.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multi-branch selection fusion fine-grained classification method based on coordinate attention positioning. Background Art

[0002] Fine-grained visual classification aims to distinguish various finer-grained object categories through subtle and indistinguishable visual differences between subordinate categories of the same class of objects. Due to the influence of deformation, occlusion, care, shooting angle, etc., there are usually only indistinguishable visual differences between different subcategories, which makes fine-grained visual classification a challenging task.

[0003] Convolutional neural networks still dominate image classification, but existing single convolutional neural networks perform poorly in fine-grained visual classification.

[0004] Therefore, it is necessary to provide a multi-branch selection fusion fine-grained classification method based on coordinate attention positioning to improve the efficiency and accuracy of fine-grained visual classification. Summary of the invention

[0005] The present invention provides a multi-branch selection fusion fine-grained classification method based on coordinate attention positioning, comprising: establishing a multi-branch selection fusion fine-grained classification model, wherein the multi-branch selection fusion fine-grained classification model comprises an original branch, a selection branch distribution space pool pyramid, a region-to-coordinate attention object positioning module, an object branch, a first selection branch block, an attention part suggestion module and a part branch, wherein the original branch is used to extract the overall features of the original image, the selection branch distribution space pool pyramid and the region-to-coordinate attention object positioning module are used to generate a target image according to the overall features of the original image, the object branch is used to extract the features of the target image, the first selection branch block is used to fuse the overall features of the original image and the features of the target image to generate fused features, the attention part suggestion module is used to crop the target image into multiple part images according to the fused features, and the part branch is used to generate classification results according to the multiple part images; training the multi-branch selection fusion fine-grained classification model; obtaining an image to be classified; and classifying the image to be classified by the trained multi-branch selection fusion fine-grained classification model.

[0006] Furthermore, the selection branch distribution spatial pool pyramid includes multiple convolutional layers and a second selection branch block, wherein the convolutional layer is used to sample the overall features of the original image, the sampling intervals of the multiple convolutional layers are continuous, and the second selection branch block is used to fuse the output results of the multiple convolutional layers.

[0007] Furthermore, the convolution layer samples the overall features of the original image based on the following formula: Among them, y[i] is the output of the convolution layer, x[i] is the input of the convolution layer, r is the sampling interval of the convolution layer, is a filter with a coefficient of k, and K is the total number of filter coefficients.

[0008] Furthermore, the selection branch distribution spatial pool pyramid and region-pair coordinate attention object positioning module generate a target image according to the overall features of the original image, including: the region-pair coordinate attention object positioning module generates an activation feature map according to the output of the second selection branch block; divides the activation feature map into multiple regions; calculates the average value of the activation feature map; obtains a mask map of each region according to the average value of the activation feature map; and generates the target image according to the mask map of each region.

[0009] Further, the average value of the activation feature map is calculated according to the following formula, including: in, is the average value of the activation feature map, W is the width of the activation feature map, H is the height of the activation feature map, and A(i,j) is the activation value of the i-th row and j-th column of the activation feature map.

[0010] Furthermore, the selection branch distribution spatial pooling pyramid and region pair coordinate attention object positioning module generate a target image according to the overall features of the original image, including: the region pair coordinate attention object positioning module uses two spatial scale pooling kernels to obtain an overall representation of the horizontal and vertical information of the output of the second selection branch block; calculates the vertical attention weight and the horizontal attention weight; obtains the horizontal mask map and the vertical mask map according to the feature map corresponding to each channel, the vertical attention weight, the horizontal attention weight and the output of the second selection branch block; generates the target image according to the intersection of the horizontal mask map and the vertical mask map.

[0011] Furthermore, the first selection branch block fuses the overall features of the original image and the features of the target image to generate fused features, including: performing element-wise summation on the overall features of the original image and the features of the target image to obtain all branch information; performing global average pooling on all branch information to generate global information; generating a feature descriptor based on the global information; generating an attention weight of each spatial dimension of the overall features of the original image and an attention weight of each spatial dimension of the features of the target image based on the feature descriptor, the attention vector of the overall features of the original image and the attention vector of the features of the target image; generating a fused feature based on the overall features of the original image, the features of the target image, the attention weight of each spatial dimension of the overall features of the original image and the attention weight of each spatial dimension of the features of the target image.

[0012] Furthermore, the attention weight of each spatial dimension corresponding to the overall feature of the original image and the attention weight of each spatial dimension corresponding to the feature of the target image are generated based on the following formula, including: Among them, a c is the attention weight of the overall feature of the original image corresponding to the cth spatial dimension, b c is the attention weight of the feature of the target image corresponding to the cth spatial dimension, A is the attention vector of the overall feature of the original image, B is the attention vector of the feature of the target image, and z is the feature descriptor.

[0013] Furthermore, training the multi-branch selection fusion fine-grained classification model includes: training the model using a training set; calculating the loss of the multi-branch selection fusion fine-grained classification model according to a loss function; and optimizing the multi-branch selection fusion fine-grained classification model according to the loss of the multi-branch selection fusion fine-grained classification model until preset conditions are met.

[0014] Furthermore, the loss function is: total =α*L raw +β*L object +γ*L parts , where L total is the total loss, α, β, γ are hyper parameters, L raw is the original branch loss, L abject is the object branch loss, L patrs is the part branch loss.

[0015] Compared with the prior art, the multi-branch selection fusion fine-grained classification method based on coordinate attention positioning provided by the present invention has at least the following beneficial effects:

[0016] A multi-branch selective fusion fine-grained classification model including the original branch, object branch and part branch is established. The image is directly input into the original branch. The Coordinate Attention Object Localization Module (CAOLM) is used to locate and crop the objects in the image and generate the input of the object branch. The Attention Partial Proposal Module (APPM) is used to propose local regions of different scales. These three types of input images are subjected to end-to-end weakly supervised learning through different branches of the network. The model expands the receptive field to capture multi-scale features by selecting the branch distribution spatial pooling pyramid (SB-ASPP). It can fuse the feature maps obtained from the original branch and the target branch with the selective branch block, and use the complete features of the original branch to supplement the missing information of the target branch, effectively improving the efficiency and accuracy of fine-grained visual classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0018] Figure 1 It is a flowchart of a multi-branch selection fusion fine-grained classification method based on coordinate attention positioning shown in some embodiments of this specification;

[0019] Figure 2 It is a schematic diagram of the structure of a multi-branch selection fusion fine-grained classification model according to some embodiments of this specification;

[0020] Figure 3 is a schematic diagram of the structure of a selection branch distribution space pool pyramid according to some embodiments of this specification;

[0021] Figure 4 is a schematic diagram of the structure of a region pair coordinate attention object positioning module according to some embodiments of this specification;

[0022] Figure 5 It is a structural schematic diagram of a first selection branch block according to some embodiments of this specification. DETAILED DESCRIPTION

[0023] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0024] Step 110, establishing a multi-branch selection fusion fine-grained classification model.

[0025] Figure 2 is a schematic diagram of the structure of a multi-branch selection fusion fine-grained classification model according to some embodiments of this specification, such as Figure 2 As shown, the multi-branch selection fusion fine-grained classification model includes an original branch, a selection branch distribution space pool pyramid, a region pair coordinate attention object positioning module, an object branch, a first selection branch block, an attention part proposal module and a part branch.

[0026] The original branch is used to extract the overall features of the original image.

[0027] The branch distribution spatial pooling pyramid and region pair coordinate attention object localization module are selected to generate a target image based on the overall features of the original image. The target image not only contains all the structural features of the target, but also contains more fine-grained features and cuts out the background that is not helpful for classification.

[0028] The object branch is used to extract the features of the target image, and the first selection branch block is used to fuse the overall features of the original image and the features of the target image to generate fused features.

[0029] The attention part proposal module is used to crop the target image into multiple part images according to the fused features, and the part branch is used to generate classification results according to the multiple part images. Specifically, the attention part proposal module crops multiple different parts with the largest difference and the smallest mutual inclusion from the target image as part images.

[0030] Specifically, the original branch is used to extract the overall features of the original image. The original branch has three output branches. The first output branch directly outputs the classification result of the original branch through the fully connected layer; in the second output branch, the overall features of the extracted original image and the output result of the last convolutional layer of the object branch are sent to the first selective branch block together, and then the classification result of the object branch is output through the fully connected layer; in the third output branch, the overall features of the extracted original image are sent to the region pair coordinate attention object positioning module after passing through the branch distribution space pool pyramid to generate a mask map for cropping the original image. The cropped target image is used as the input of the object branch. The object branch has two output branches, one output branch is used to send the features of the extracted target image to the first selective branch block, and the other output branch is used to send the features of the extracted target image to the attention part proposal module. The cropped multiple part images are used as the input of the part branch.

[0031] Figure 3 is a schematic diagram of a structure of a selection branch distribution space pool pyramid according to some embodiments of this specification, such as Figure 3 As shown, in some embodiments, the selection branch distribution spatial pooling pyramid includes multiple convolutional layers and a second selection branch block, wherein the convolutional layer is used to sample the overall features of the original image, the sampling intervals of the multiple convolutional layers are continuous, and the second selection branch block is used to fuse the output results of the multiple convolutional layers.

[0032] Specifically, the branch distribution spatial pooling pyramid is used to perform parallel sampling of the input signal at multiple different sampling intervals in the same layer, which can accurately control the spatial resolution and capture features of different scales. In order to locate the object before the region coordinate attention object localization module locates the object, the multi-branch selection fusion fine-grained classification model can capture the features of different scales in different parts of the object. The branch distribution spatial pooling pyramid is still based on attribute convolution to adjust the response in one layer at any resolution.

[0033] In some embodiments, the convolutional layer samples the overall features of the original image based on the following formula:

[0034]

[0035] Among them, y[i] is the output of the convolution layer, x[i] is the input of the convolution layer, r is the sampling interval of the convolution layer, is a filter with a coefficient of k, and K is the total number of filter coefficients.

[0036] The convolution results obtained in a certain layer come from an independent set of the previous layer and have no mutual dependence, because the calculation method of the Aless convolution is close to the checkerboard format. In this layer, there is no correlation between the convolution results, that is, local information is lost. At the same time, due to the sparse sampling input signal of the long-distance convolution, the information obtained by the long-distance convolution has no correlation, which affects the classification results. In order to solve the problem of local information loss and lack of correlation, the second selection branch block is used to fuse convolutions of different strides, and the convolution output results of different strides can complement each other with missing information. When setting the convolution stride, in order to avoid information loss, the HDC (Hybrid Dilated Convolution) processing method is introduced, and the stride is set to be continuous. For example, three sampling intervals of r=1, r=2 and r=3 are set.

[0037] In some embodiments, the branch distribution spatial pooling pyramid and the region pair coordinate attention object localization module are selected to generate a target image according to the overall features of the original image, including:

[0038] The region pair coordinate attention object localization module generates an activation feature map based on the output of the second selection branch block;

[0039] Divide the activation feature map into multiple regions;

[0040] Calculate the average of the activation feature maps;

[0041] According to the average value of the activated feature map, the mask map of each region is obtained;

[0042] Based on the mask map of each region, a target image is generated.

[0043] Specifically, the region pair coordinate attention object localization module obtains the activation map by aggregating the feature maps in the channel dimension. The activation map is used to generate a bounding box to obtain the target image. The method for obtaining the activation map is optimized, and the specific process is as follows. C×H×W is the output of the second selection branch block, the channel number is C, the spatial size is H×W, W is the width, and H is the height. For example, the activation feature map A is divided into four equal regions, with H / 2 and W / 2 as the boundaries. Use the average value of the activation feature map The threshold is used to determine whether each area belongs to the key area in the recognition task.

[0044] In some embodiments, the average value of the activation feature map is calculated according to the following formula, including:

[0045]

[0046] in, is the average value of the activation feature map, W is the width of the activation feature map, H is the height of the activation feature map, and A(i,j) is the activation value of the i-th row and j-th column of the activation feature map.

[0047] The mask map is obtained according to the following formula. Finally, the mask map of each region is spliced ​​together according to the corresponding activation feature map A to obtain the final mask map.

[0048]

[0049] Where, is the mask map of the i-th row and j-th column of the activation feature map.

[0050] In some embodiments, the branch distribution spatial pooling pyramid and the region pair coordinate attention object localization module are selected to generate a target image according to the overall features of the original image, including:

[0051] The region pair coordinate attention object localization module uses two spatial scale pooling kernels to obtain an overall representation of the horizontal and vertical information of the output of the second selection branch block:

[0052] Calculate vertical attention weights and horizontal attention weights;

[0053] According to the feature map corresponding to each channel, the vertical attention weight, the horizontal attention weight and the output of the second selection branch block, a horizontal mask map and a vertical mask map are obtained;

[0054] Generate the target image based on the intersection of the horizontal mask map and the vertical mask map.

[0055] Figure 4 is a schematic diagram of the structure of the region coordinate attention object positioning module according to some embodiments of this specification, such as Figure 4 As shown in Figure 2, specifically, the purpose of the region-to-coordinate attention object localization module is to give a greater weight to the region where the identified object is located in space. C×H×W Represents the output of the region pair coordinate attention object localization module. Use two spatial scale pooling kernels (H, 1) and (1, W) to obtain the overall representation of horizontal and vertical information respectively. Calculate the horizontal coordinate attention part and the vertical coordinate attention part on the two branches respectively. The c-th channel output by different pooling layers on different branches can be expressed as:

[0056]

[0057] in, is the output of vertical average pooling (X Avg Pool) at the cth channel, is the output of the vertical maximum pooling (X Max Pool) in the cth channel, is the output of the horizontal maximum pooling (Y AvgPool) at the cth channel, is the output of the horizontal maximum pooling (Y Max Pool) at the cth channel, x c (h, i) is the i-th level information, x c (j, w) is the jth vertical information.

[0058] Taking vertical branch or horizontal branch as an example, z havg (h) and z hmax (h) The resulting z h (h) The 1×1 convolution function F that shares parameters between the vertical branch and the horizontal branch is fed into 1 :

[0059]

[0060] Among them, f h ∈R C / r×2H is an intermediate feature map that encodes spatial information in the vertical direction. δ represents a nonlinear activation function.

[0061] f h Split into and

[0062] Taking the average pooling branch of horizontal encoding as an example, another 1×1 convolution transformation function F h ( and Share the parameters of F) scaled Transformed into a vertical average attention weight g with the same number of channels as the input X havg ,

[0063]

[0064] Among them, σ represents the sigmoid function. Similarly, the horizontal average attention weight g can be obtained wavg , the vertical maximum attention weight g hmax and the horizontal maximum attention weight g wmax .

[0065] The feature map y obtained using different downsampling methods is obtained based on the following formula avg and max Among them, y avg and max is the feature map corresponding to the output feature map channel c:

[0066]

[0067] y avg ,ymax The output of the second selective branch block is input into the third selective branch block to obtain the output. The final mask map M of the region pair coordinate attention object localization module is the mask map of the horizontal branch And the mask map of the vertical branch The intersection of .

[0068] Figure 5 is a schematic diagram of the structure of the first selection branch block according to some embodiments of this specification, such as Figure 5 As shown, in some embodiments, the first selection branch block fuses the overall features of the original image and the features of the target image to generate fused features, including:

[0069] The overall feature M1 of the original image and the feature M2 of the target image are summed element by element, and the information U from all branches is integrated using the element sum of M1 and M2;

[0070] To generate s∈R C Channel-level statistics, using global average pooling on U to generate global information, where the c-th element of s is the reduction of the c-th spatial dimension H×W corresponding to the feature map U;

[0071] Generate feature descriptors based on global information;

[0072] Based on the feature descriptor, the attention vector of the overall feature of the original image and the attention vector of the feature of the target image, the attention weight of the overall feature of the original image corresponding to each spatial dimension and the attention weight of the feature of the target image corresponding to each spatial dimension are generated;

[0073] Based on the overall features of the original image, the features of the target image, the attention weights of the overall features of the original image corresponding to each spatial dimension, and the attention weights of the features of the target image corresponding to each spatial dimension, a fused feature is generated.

[0074] Specifically, statistical information can be generated based on the following formula:

[0075]

[0076] Among them, s c is the information of the cth spatial dimension, F gp (U c ) is to use global average pooling for U.

[0077] In order to guide the adaptive selection of the next feature in the subsequent selection operation, the feature descriptor z∈R d+1 Generated by the fully connected layer. This method reduces the feature dimension and improves computational efficiency. The feature descriptor z is generated according to the following formula:

[0078] z=Ffc (s) = δ(ξ(Ws))

[0079] Among them, δ represents the ReLU function, ξ represents batch normalization, and W∈R dxC In order to reduce the impact of d on model efficiency, the reduction ratio r is used to control its value:

[0080]

[0081] Wherein, L represents the minimum value of d.

[0082] In some embodiments, a Softmax operation is performed on the data in the channel direction, and the attention weight corresponding to each spatial dimension of the overall feature of the original image and the attention weight corresponding to each spatial dimension of the feature of the target image are generated based on the following formula, including:

[0083]

[0084] Among them, a c is the attention weight of the overall feature of the original image corresponding to the cth spatial dimension, b c is the attention weight of the cth spatial dimension corresponding to the feature of the target image, A is the attention vector of the overall feature of the original image, B is the attention vector of the feature of the target image, and z is the feature descriptor. A,B∈R Cxd and a, b represent the attention vectors of M1 and M2 respectively. It is worth noting that A c ∈R 1xd is the cth row of vector A, a c represents the cth element of vector a, for B c and b c This is also true. Because a c +b c = 1, the matrix B is redundant in the case of two branches. The output V is obtained by the attention weight of each kernel:

[0085]

[0086] Where V = [V 1 ; V 2 ; V 3 ;…;V c ]; V c ∈R HxW .

[0087] Step 120: training a multi-branch selection fusion fine-grained classification model.

[0088] In some embodiments, training a multi-branch selection fusion fine-grained classification model includes:

[0089] Use the training set to train the model;

[0090] According to the loss function, calculate the loss of the multi-branch selection fusion fine-grained classification model;

[0091] According to the loss of the multi-branch selection fusion fine-grained classification model, the multi-branch selection fusion fine-grained classification model is optimized until the preset conditions are met.

[0092] In some embodiments, the loss function is:

[0093] L total =α*L raw +β*L object +γ*L parts

[0094] Among them, L total is the total loss, α, β, γ are hyper parameters, L raw is the original branch loss, L object is the object branch loss, L patrs is the part branch loss. As an example, α=0.75, β=1, γ=0.75.

[0095] Cross entropy loss is used as the classification loss for all three branches:

[0096] L raw = -log(P r (c))

[0097] L object = -log(P o (c))

[0098]

[0099] Where c represents the true label of the image. r (c) is the classification probability generated by the last Softmax layer of the original branch. o (c) and Generated in the same way from the object branch and the part branch respectively. n represents the nth part image, and N is the total number of cropped regions in each image.

[0100] Step 130: Obtain an image to be classified.

[0101] Step 140, classify the image to be classified by using the trained multi-branch selection fusion fine-grained classification model.

[0102] These three branches work together in the back-propagation process to optimize the performance of the model, so that the model can use the overall structural features of the object and the fine-grained features of the parts to obtain better classification prediction accuracy. The Selective Branch Block (SBBlock) strengthens the connection between branches, and uses the original branch to have complete features of the object. When the Coordinate Attention Object Localization Module (CAOLM) is not accurate in the region positioning, it supplements the missing information of the object branch. The Selective Branch-rous Spatial Pooling Pyramid (SB-ASPP) can not only capture multi-scale information, but also alleviate the problem of missing information in the Selective Branch-rous Spatial Pooling Pyramid (SB-ASPP). The multi-branch selection fusion fine-grained classification model has good target scale adaptability, and when more attention is paid to those discriminative features during the positioning process, SB-ASPP and CAOLM improve the robustness of positioning training. All modules used in the multi-branch selection fusion fine-grained classification model are designed to be lightweight and efficient, and during the testing phase, some branches are removed to reduce a lot of calculations. Because of this, the multi-branch selection fusion fine-grained classification model takes a very short time to predict in practical applications. Due to the reasonable and efficient architecture, the multi-branch selection fusion fine-grained classification model outperforms almost all methods based on convolutional neural network architectures on the datasets used in the experiments.

[0103] The following experiments are combined to verify the performance of the multi-branch selection fusion fine-grained classification model.

[0104] The image size is fixed to 448×448 to obtain the input of the original branch. The target image is also scaled to 448×448, but for the part branch, the size of all part images is 224×224. In order to alleviate overfitting, random flipping and Mixup (mixing parameter is 0.1) are used as image augmentation strategies. For the activation map of size 14×14, three scales of windows are used: {[4×4,3×5],[6×6,5×7],[8×8,6×10,7×9,7×10]}. The number of part images of the original image is N=7, where N 1 =2, N 2 =3 and N 3=2. N1, N2, and N3 are the number of scale windows of the above three categories. The model backbone uses ResNext-50 pre-trained on ImageNet. No additional annotation information is used during training and testing. Stochastic gradient descent is used as the optimizer with a momentum of 0.9% and a weight decay of 0.0001. On an NVIDIA V100 GPU, the batch size of the model is 6, the initial learning rate is 0.001, and it is multiplied by 0.1 after 60 epochs.

[0105] The dataset 1 used in the experiment contains a total of 11,788 annotated images, of which 5,909 are used for training and 5,879 are used for testing. The dataset images contain a total of 200 different categories of birds. The dataset 2 used in the experiment contains a total of 16,185 annotated images, of which 8,144 are used for training and 8,041 are used for testing. The dataset images contain a total of 196 cars of different brands and models. The dataset 3 used in the experiment contains a total of 10,000 annotated images, of which 6,667 are used for training and 3,333 are used for testing. These images are usually divided into 100 different types of aircraft in fine-grained visual classification. The classification tasks on these datasets remain challenging.

[0106] The experimental results are shown in Table 1.

[0107] Table 1

[0108]

[0109] As shown in Table 1, our method is compared with the state-of-the-art algorithms on Dataset 1, Dataset 2, and Dataset 3 without using additional annotations. From the results, our method steadily outperforms similar methods that crop images through multi-branch cyclic training. The classification performance of MMAL-Net using ResNext-50 as the backbone network is significantly reduced, demonstrating the versatility of CAOLM for backbone networks. On Dataset 1, CAP achieves state-of-the-art classification performance. It proposes a context-aware attention pooling (CAP) module that can effectively encode the location and appearance information of local features. The module takes the features output by the convolutional network as input and learns to adjust the importance of different regions in the features, thereby obtaining rich appearance features and spatial features of local regions, and then accurately classifies them. On Dataset 2, our model has similar classification performance to CAP, but the performance of our model is significantly improved compared to the backbone used. When it uses the same backbone as our model, CAP has an accuracy of only 94.9% on Dataset 2. Among all the algorithms used for comparison, our model achieves the best classification performance on Dataset 3. The only method with similar classification performance uses Inceptionv4 as the backbone, and it uses a complex training strategy instead of end-to-end training. In addition, the performance of this model on Dataset 3 is significantly better than other models. Dataset 3 improves by 10.7% and 15.4% over the baseline (ResNext-50) on Dataset 2 and Dataset 3, respectively. When ResNet-50 is used as the baseline, there are also 10.6% and 15.4% improvements on these two datasets, respectively. This is attributed to the precise object localization of CAOLM and the use of different branch information by SBBlock to complement each other. SB-ASPP avoids the information loss problem caused by void convolution while capturing multi-scale information.

[0110] Ablation experiments were performed on Dataset 3.

[0111] a) Effectiveness of the three-branch structure: In this model (using ResNext-50 as the backbone network), if only the original branch is used, the classification accuracy is 85.3%. On this basis, if the model has an object branch, the classification accuracy can reach 93.5%. The three-branch model can achieve a classification accuracy of 95.7%. Experimental results show that if the part branch is not used in the reasoning stage, the model can double the reasoning speed, while the classification results are still excellent. The classification accuracy of the model in this case is 95.6%.

[0112] b) Effectiveness of CAOLM: When ResNet-50 is used as the backbone in MMALNet, an accuracy of 94.7% can be achieved. However, when the backbone is replaced with ResNet-50, the accuracy of MMAL-Net drops to 93.3%. When the convolutional neural network focuses too much on discriminative features, the performance of AOLM will deteriorate. Also using ResNet-50 as the backbone network, replacing AOLM in MMAL-Net with CAOLM, the accuracy is improved to 95.1%. When SBBlock is used, the accuracy of MMAL-Net is improved to 95.3%. It is proved that this method can effectively improve the positioning and classification performance.

[0113] c) Effectiveness of replacing the backbone network: The backbone network used in this method is ResNext-50, which achieves 95.6% classification accuracy on Dataset 3. This is better than the classification accuracy (95.3%) when ResNet-50 is used as the backbone network. This proves that CAOLM has better versatility for the backbone network of the model.

[0114] d) Effectiveness of SB-ASPP: If SB-ASPP is removed, the accuracy drops from 95.6% to 95.0%. SB-ASPP can adjust the network receptive field and focus on global multi-scale information, alleviating the disadvantage that the AOLM localization performance will decrease when the convolutional neural network focuses on discriminative features very effectively.

[0115] Compared with the true bounding box, the correctly positioned predicted box ratio IOU (Intersection over Union) exceeds 50% for PCP (Percent of Right Localized Parts). As shown in Table 2, the positioning ability of CAOLM is significantly improved compared with SCDA and AOL. Although multi-layer fusion significantly improves the performance of AOLM, it is found in actual experiments that it takes a lot of experiments to find which two layers to use for fusion in different networks. This method reduces the universality of AOLM, and the single-layer AOLM is not ideal in actual experiments. The positioning accuracy of AOLM can reach up to 85.1%, but it actually drops to 71.1% with the training of the network. It is worth mentioning that if the Mixup image enhancement strategy is adopted, AOLM cannot find the intersection of the two layers of activation maps, which proves that AOLM is not robust to input signal noise. The accuracy of CAOLM can reach up to 87.4%, and the accuracy drops to 77.1% after network training, while the accuracy rises to 79.1% after using SB-ASPP. APPM partially branches the cropped images so that the model can adapt to the scale of the object while still achieving good classification results.

[0116] Table 2

[0117] algorithm Accuracy SCDA 76.8％ AOLM 85.1％ CALOM 87.4％

[0118] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A multi-branch selection fusion fine-grained classification method based on coordinate attention positioning, characterized in that: include: Establish a multi-branch selection fusion fine-grained classification model, wherein the multi-branch selection fusion fine-grained classification model includes an original branch, a selection branch distribution space pool pyramid, a region-to-coordinate attention object positioning module, an object branch, a first selection branch block, an attention part proposal module and a part branch, wherein the original branch is used to extract the overall features of the original image, the selection branch distribution space pool pyramid and the region-to-coordinate attention object positioning module are used to generate a target image according to the overall features of the original image, the object branch is used to extract the features of the target image, the first selection branch block is used to fuse the overall features of the original image and the features of the target image to generate fused features, the attention part proposal module is used to crop the target image into multiple part images according to the fused features, and the part branch is used to generate classification results according to the multiple part images; Training the multi-branch selection fusion fine-grained classification model; Get the image to be classified; The image to be classified is classified by using the trained multi-branch selection fusion fine-grained classification model.

2. The multi-branch selection fusion fine-grained classification method based on coordinate attention positioning according to claim 1 is characterized in that: The selection branch distribution spatial pool pyramid includes multiple convolutional layers and a second selection branch block, wherein the convolutional layer is used to sample the overall features of the original image, the sampling intervals of the multiple convolutional layers are continuous, and the second selection branch block is used to fuse the output results of the multiple convolutional layers.

3. The multi-branch selection fusion fine-grained classification method based on coordinate attention positioning according to claim 2 is characterized in that: The convolutional layer samples the overall features of the original image based on the following formula: Among them, y[i] is the output of the convolution layer, x[i] is the input of the convolution layer, r is the sampling interval of the convolution layer, is a filter with a coefficient of k, and K is the total number of filter coefficients.

4. The multi-branch selection fusion fine-grained classification method based on coordinate attention positioning according to claim 2 is characterized in that: The selection branch distribution space pool pyramid and region pair coordinate attention object positioning module generates a target image according to the overall features of the original image, including: The region pair coordinate attention object positioning module generates an activation feature map according to the output of the second selection branch block; dividing the activation feature map into a plurality of regions; Calculating an average value of the activation feature map; Obtaining a mask map for each region according to the average value of the activation feature map; The target image is generated according to the mask mapping of each region.

5. The multi-branch selection fusion fine-grained classification method based on coordinate attention positioning according to claim 4 is characterized in that: The average value of the activation feature map is calculated according to the following formula, including: in, is the average value of the activation feature map, W is the width of the activation feature map, H is the height of the activation feature map, and A(i,j) is the activation value of the i-th row and j-th column of the activation feature map.

6. The multi-branch selection fusion fine-grained classification method based on coordinate attention positioning according to claim 4 is characterized in that: The selection branch distribution space pool pyramid and region pair coordinate attention object positioning module generates a target image according to the overall features of the original image, including: The region pair coordinate attention object localization module uses two spatial scale pooling kernels to obtain an overall representation of the horizontal and vertical information of the output of the second selection branch block: Calculate vertical attention weights and horizontal attention weights; Obtain a horizontal mask map and a vertical mask map according to the feature map corresponding to each channel, the vertical attention weight, the horizontal attention weight, and the output of the second selection branch block; The target image is generated according to the intersection of the horizontal mask map and the vertical mask map.

7. The multi-branch selection fusion fine-grained classification method based on coordinate attention positioning according to any one of claims 1 to 6, characterized in that: The first selection branch block fuses the overall features of the original image and the features of the target image to generate fused features, including: Performing element-wise summation on the overall features of the original image and the features of the target image to obtain all branch information; Performing global average pooling on all branch information to generate global information; Based on the global information, generating a feature descriptor; Based on the feature descriptor, the attention vector of the overall feature of the original image and the attention vector of the feature of the target image, generating an attention weight corresponding to each spatial dimension of the overall feature of the original image and an attention weight corresponding to each spatial dimension of the feature of the target image; Based on the overall features of the original image, the features of the target image, the attention weights of the overall features of the original image corresponding to each spatial dimension, and the attention weights of the features of the target image corresponding to each spatial dimension, a fused feature is generated.

8. The multi-branch selection fusion fine-grained classification method based on coordinate attention positioning according to claim 7 is characterized in that: The attention weight of each spatial dimension corresponding to the overall feature of the original image and the attention weight of each spatial dimension corresponding to the feature of the target image are generated based on the following formula, including: Among them, a c is the attention weight of the overall feature of the original image corresponding to the cth spatial dimension, b c is the attention weight of the feature of the target image corresponding to the cth spatial dimension, A is the attention vector of the overall feature of the original image, B is the attention vector of the feature of the target image, and z is the feature descriptor.

9. The multi-branch selection fusion fine-grained classification method based on coordinate attention positioning according to any one of claims 1 to 6, characterized in that: Training the multi-branch selection fusion fine-grained classification model includes: Use the training set to train the model; According to the loss function, calculating the loss of the multi-branch selection fusion fine-grained classification model; According to the loss of the multi-branch selection fusion fine-grained classification model, the multi-branch selection fusion fine-grained classification model is optimized until a preset condition is met.

10. The multi-branch selection fusion fine-grained classification method based on coordinate attention positioning according to claim 9 is characterized in that: The loss function is: L total =α*L raw +β*L object +γ*L parts Among them, L total is the total loss, α, β, γ are hyper parameters, L raw is the original branch loss, L object is the object branch loss, L patrs is the part branch loss.

Images