Multi-scale salient feature double-line attention fine-grained object classification method and system

Abstract

The application provides a multi-scale salient feature double-line attention fine-grained object classification method and system, and the specific steps are as follows: a multi-scale salient feature double-line attention fine-grained classification network is constructed; a classification label, a double-line attention pooling strategy output and a double-line attention pooling strategy output are used to construct a loss function; the multi-scale salient feature double-line attention fine-grained classification network is trained to obtain a multi-scale salient feature double-line attention fine-grained classification network model; and the preprocessed training data set is sent into the multi-scale salient feature double-line attention fine-grained classification network model to obtain a fine-grained classification result. The multi-scale salient feature double-line attention fine-grained object classification network model solves the problem that the existing fine-grained image classification of objects in the same field has too fine distinguishing features that are difficult to capture and cannot effectively locate the region of interest, resulting in inaccurate classification results.

Description

Technical Field

[0001] This invention belongs to the field of electronic information technology, specifically to a multi-scale salient feature dual-line attention fine-grained object classification method and system. Background Technology

[0002] In computer vision tasks, deep learning technology has achieved excellent performance in many branches of image processing in recent years, even surpassing human recognition accuracy in image classification tasks. However, compared to image recognition technology, there is still considerable room for improvement in the accuracy of fine-grained recognition. Classification tasks can be divided into coarse-grained classification and fine-grained classification. Coarse-grained tasks represent categories with significant differences, such as classifying cats and dogs, or cars and airplanes. Fine-grained classification focuses on classifying subtle visual differences within basic categories, such as classifying dogs, airplanes, cars, and birds into more detailed subcategories. Fine-grained classification has a wide range of applications, such as in the identification of wild flora and fauna, providing an important technological foundation for the field of biology; it can be applied to visual tasks such as clothing detection and recognition; it can be used for automated checkout services in retail scenarios; and it can be used for fast, accurate, and precise identification of high-speed vehicles.

[0003] The key to solving fine-grained image classification is enabling neural networks to locate distinguishable parts and learn discriminative features. Early approaches typically employed a two-stage, strongly supervised method: first, a neural network locates and detects distinguishable components in the image and extracts discriminative local features; then, a classification neural network performs fine-grained image classification based on these local features. However, this two-stage strongly supervised approach relies heavily on labeled bounding boxes and component annotations. Furthermore, fine-grained images are difficult for the average person to distinguish, requiring additional component annotations from professionals, resulting in significant manpower costs and making these methods impractical.

[0004] In recent years, researchers have focused more on weakly supervised methods that only use image-level labels, i.e., using only image-level label information during training. Attention mechanisms can guide models to focus on more discriminative parts, improving the accuracy of fine-grained classification tasks. While attention mechanisms have shown great potential in fine-grained tasks, they lack attention to subtle features, easily overlooking potential discriminative information. Furthermore, attention models lack effective supervision to guide the learning process, thus failing to accurately locate effective local regions in many situations, leading to low accuracy in fine-grained classification of objects in different application scenarios. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a multi-scale salient feature dual-line attention fine-grained object classification method and system. It uses a region salient feature enhancement module to highlight the most salient part of the current stage feature map and amplify the subtle features of the initial feature map. Through a dual-line attention pooling strategy, an attention map is generated from the enhanced feature maps at different stages to represent the salient features of the object. Furthermore, counterfactual attention learning guides the attention learning process, encouraging the attention mechanism to discover the most discriminative regions, thus achieving accurate differentiation of objects within the same domain.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-scale salient feature bilinear attention fine-grained object classification method, the specific steps of which are as follows:

[0007] S1 constructs a multi-scale salient feature dual-line attention fine-grained classification network. The network includes a basic backbone network ResNet50. After Stage 3, Stage 4 and Stage 5 of the basic backbone network ResNet50, a Region Satisfactory Feature Enhancement Module (RPFBM) is connected. After the RPFBM, a dual-line attention pooling strategy (BAP) is connected. Both the RPFBM and the BAP are connected to the Counterfactual Attention Learning Network (CAL). The CAL contains the BAP.

[0008] S2. Construct a loss function using the classification label y, the output of the dual-line attention pooling strategy BAP, and the output of the dual-line attention pooling strategy BAP.

[0009] S3 is used to train a multi-scale salient feature bilinear attention fine-grained classification network to obtain a multi-scale salient feature bilinear attention fine-grained classification network model.

[0010] S4 feeds the preprocessed training dataset into the multi-scale salient feature bilinear attention fine-grained classification network model to obtain fine-grained classification results of objects.

[0011] Furthermore, in S1, the basic backbone network ResNet50 is used to obtain feature maps X∈F at different scales. C×W×H Where C, W, and H represent the number of channels, width, and height of the feature map, it contains 5 stages, where stage 1 is used for preprocessing the input image, and stages 2 to 4 consist of residual blocks (Bottleneck).

[0012] Furthermore, in S1, the construction of the Region Satisfactory Feature Enhancement Module (RPFBM) specifically includes:

[0013] 1) Initial feature maps X∈F of the outputs of Stage 3, Stage 4, and Stage 5 of the basic backbone network ResNet50. C ×W×H Perform uniform slicing operations along the width and height dimensions (k and l parts respectively) to obtain N = k × l segmented feature maps X. i，j ∈F C ×(W / k)×(H / l) ,i∈[1,k], j∈[1,l];

[0014] 2) The segmented feature map X is calculated using 1*1 convolution operation and global average GAP pooling. i，j Weighting factor b i，j And weight factor b i，j The importance weight matrix B is reconstructed by dividing the matrix into blocks, where B(i,j)=b′. i，j ;

[0015] 3) Multiply the importance weight matrix B and the initial feature map X element by element, and then sum them together to generate the salient feature map X. b .

[0016] Furthermore, in S1, the construction of the dual-path attention pooling strategy BAP specifically includes:

[0017] 1) The salient feature map X obtained by the Region Salient Feature Enhancement Module (RPFBM) b Perform a 3x3 convolution operation to obtain attention map A;

[0018] 2) Attention map A and feature map X b Perform element-wise tensor multiplication to obtain the local feature map F. k k = 1, 2, ..., M, the feature extraction function GAP is used to extract the feature map F of the location. k The attentional features f of each specific region are obtained through processing. k ∈R 1×C ;

[0019] 3) Incorporate attention features f k Stacked into feature matrices P∈R at different scales M×C The feature matrix of the part P∈R M×C After passing through the fully linked layer, the fact prediction P is generated. T :

[0020]

[0021] P r =C([Γ(A,X) b )]).

[0022] Furthermore, in S1, the construction of the counterfactual attention learning network CAL specifically includes:

[0023] 1) Generate a fake attention map with the same size as attention map A. Fake attention map All values ​​in the data are random numbers;

[0024] 2) Regarding spurious attention maps A dual-path attention pooling (BAP) strategy is applied, and the result is passed through a fully connected layer to obtain the counterfactual prediction P. F :

[0025]

[0026] Among them, X b This is the salient feature map output by the Region Salient Feature Enhancement Module (RPFBM).

[0027] Furthermore, in S2, the output of the dual-track attention pooling strategy BAP is the fact prediction P. T The dual-line attention pooling strategy BAP outputs the difference result Y. effect The factual prediction P will be determined separately. T and difference results Y effect The cross-entropy is calculated with the classification label y, and the results are summed to obtain the loss function L:

[0028] L = L ce (Y effect ,y)+L ce (P T ,y).

[0029] Furthermore, in S3, the Adam optimization algorithm is used to train a multi-scale salient feature bilinear attention fine-grained classification network to obtain the final multi-scale salient feature bilinear attention fine-grained classification network model.

[0030] This invention also provides a multi-scale salient feature bilinear attention fine-grained object classification system, specifically including:

[0031] The network construction module is used to construct a multi-scale salient feature bilinear attention fine-grained classification network. The network includes a basic backbone network ResNet50. After Stage 3, Stage 4 and Stage 5 of the basic backbone network ResNet50, a Region Satisfactory Feature Enhancement Module (RPFBM) is connected. After the RPFBM, a bilinear attention pooling strategy (BAP) is connected. Both the RPFBM and the BAP are connected to the Counterfactual Attention Learning Network (CAL).

[0032] The loss function calculation module is used to establish the loss function using the classification label y, the output of the dual-line attention pooling strategy BAP, and the output of the dual-line attention pooling strategy BAP.

[0033] The model training module is used to train the multi-scale salient feature bilinear attention fine-grained classification network to obtain the multi-scale salient feature bilinear attention fine-grained classification network model.

[0034] The fine-grained classification module is used to feed the preprocessed training dataset into the multi-scale salient feature bilinear attention fine-grained classification network model to obtain the fine-grained classification results of objects.

[0035] Furthermore, the counterfactual attention learning network CAL incorporates a bilinear attention pooling strategy, BAP.

[0036] The present invention also provides a computer device, which includes a computer, a server or other terminal device with computing functions. The device includes a processor and a memory connected by a bus. The memory stores a program, and the program is executed by the processor. The program includes methods for performing the multi-scale salient feature bilinear attention fine-grained classification method as described above.

[0037] Compared with the prior art, the present invention has at least the following beneficial effects:

[0038] This invention provides a multi-scale salient feature bilinear attention fine-grained object classification method and system. It constructs a multi-scale salient feature bilinear attention fine-grained classification network, which mainly includes: a Region Satisfactory Feature Enhancement Module (RPFBM), which amplifies and captures subtle distinguishable features through region slicing operations, enhancing the expressive power of the feature map, highlighting the most salient part of the feature map at the current stage, and amplifying subtle features of the initial feature map; and a Bilinear Attention Pooling Strategy (BAP), which hierarchically represents the salient features of the object in a weakly supervised manner, improving the attention to local information at different scales, and generating an attention map from the enhanced feature maps at different stages to represent the salient features of the object. Location-based features: The Counterfactual Attention Learning Network (CAL) guides the attention learning process, encouraging the attention mechanism to discover the most discriminative regions. The difference between the learned attention and irrelevant attention in the final prediction is used as a metric. Maximizing this difference forces the bilinear attention pooling strategy to learn more effective features. The classification method of this invention incentivizes the network to autonomously explore potential discriminative information, accurately focus on local region locations, improve the network's feature representation and model generalization capabilities, and enhance the accuracy of the classification method. It solves the problems in existing fine-grained image classification tasks, such as the difficulty in capturing overly subtle discriminative features and the inability to effectively locate regions of interest.

[0039] Meanwhile, this invention calculates the cross-entropy of fact prediction and difference results with classification labels and sums them to establish a new loss function. This loss function can quantify the difference results and introduce new supervision signals to assist network training. By optimizing the loss function, the model can converge faster while improving its classification accuracy. Attached Figure Description

[0040] Figure 1 Diagram of a multi-scale salient feature bilinear attention fine-grained classification network structure;

[0041] Figure 2 Structure diagram of regional salient feature enhancement strategies;

[0042] Figure 3 Schematic diagram of the dual-line attention pooling (BAP) process;

[0043] Figure 4 Counterfactual attention learning structure diagram;

[0044] Figure 5 Feature map visualization results: (a) Original image; (b) ResNet50; (c) ResNet50+RPFBM; (d) ResNet50+RPFBM+MBAP; (e) ResNet50+RPFBM+MBAP+CAL. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0046] This invention provides a multi-scale salient feature bilinear attention fine-grained object classification method, capable of accurately classifying CUB-200-2011 (birds), Stanford Cars (cars), and Stanford Dogs (dogs) image datasets. The CUB-200-2011 (birds) dataset can be applied in ecological conservation, such as effectively identifying different bird species, providing a foundation for ecological research. The Stanford Cars dataset can be applied to urban traffic management, effectively identifying similar vehicle types and improving road traffic planning. The Stanford Dogs dataset can be used for pet dog identification, with applications in areas such as civilized dog ownership in cities. The specific process of this invention's multi-scale salient feature bilinear attention fine-grained object classification method is as follows:

[0047] 1. Download the images from the CUB-200-2011, Stanford Cars, and Stanford Dogs datasets, and perform data screening to ensure the integrity of the images. Preprocess the dataset images. In the preprocessing stage, this invention employs data augmentation methods to randomly crop, rotate, and scale the sample images, expanding the number of samples in the dataset and enhancing the robustness of the CNN model.

[0048] 2. Construct a multi-scale salient feature bilinear attention fine-grained classification network. Specific steps include:

[0049] The structure of a bilinear attention convolutional neural network based on multi-scale salient features is as follows: Figure 1 As shown, the network comprises: a basic backbone network (ResNet50), a Region Patch Feature Boosting (RPFBM) module, a Bilinear Attention Pooling (BAP) strategy, a Counterfactual Attention Learning (CAL) network, and a classifier. Specifically, the network includes the basic backbone network ResNet50. Each stage (Stage 3, Stage 4, and Stage 5) of the ResNet50 backbone network is connected to a Region Patch Feature Boosting (RPFBM) module. Each RPFBM module is followed by a Bilinear Attention Pooling (BAP) strategy. Both the RPFBM and BAP strategies are connected to the Counterfactual Attention Learning (CAL) network, which incorporates the BAP strategy.

[0050] The region saliency enhancement module takes the output of the backbone network stage 3-5 as input and operates to obtain saliency feature maps X for different branches. b And the salient feature map X b As input to the bilinear attention pooling strategy BAP, and after passing through the multi-branch bilinear attention pooling strategy MBAP, the outputs of different branches can be used as the prediction result P by the classifier input. 1T P 2T P 3T In this process, to correct the attention map's focus, a counterfactual attention learning network (CAL) is introduced to generate fake attention maps and fake prediction results P. 1F P 2F P 3F This is to correct the actual prediction results.

[0051] First, a basic backbone network (ResNet50) is constructed, consisting of five stages. Stage 1 has a simple structure and can be considered as preprocessing the input image. Stages 2 through 4 have similar structures, composed of residual blocks (Bottlenecks). The network depth increases with each stage, and the information contained therein becomes richer. After the input image passes through different stages of the backbone network, feature maps X∈F at different scales can be obtained. C×W×H Where C, W, and H represent the number of channels, width, and height of the feature map.

[0052] Secondly, for fine-grained tasks, some salient features are often contained in small regions. This invention designs a Region Salient Feature Enhancement Module (RPFBM), which optimizes the feature representation of the current stage of the network by increasing the attention weight of the region where the salient features are located, amplifies specific salient features, and promotes the network's ability to capture subtle distinguishable features.

[0053] Then, the Region Satisfactory Feature Enhancement (RPFBM) module amplifies subtle salient features, but simply increasing the attention to subtle regions is insufficient to fully capture local information. Therefore, this invention designs a dual-line attention pooling strategy (BAP) to explore the local information hidden in subtle regions of the target object through multiple dual-line attention pooling BAP processes at multiple levels.

[0054] Finally, since MBAP is only supervised by the loss function and lacks evaluation information to measure attention quality and guide the learning process, the attention maps A generated by each MBAP process are not always effective, with limited discriminative power and robustness. Therefore, this invention adds a counterfactual attention learning network, employing counterfactual learning principles to address the issue based on the generated spurious attention maps. The penalty prediction result encourages attention map A to extract the most discriminative region as much as possible, avoiding suboptimal results, and effectively solves the problems existing in MBAP mentioned above.

[0055] 3. The execution of the basic backbone network includes the following specific steps:

[0056] Images from the preprocessed training dataset are fed into ResNet50 to generate initial feature maps at different scales. These initial feature maps are labeled as X∈F. C×W×H Where C, W, and H represent the number of channels, width, and height of the feature map.

[0057] 4. Construction of the Region Satisfactory Feature Enhancement Module (RPFBM), including the following steps:

[0058] The structure of the Region Saliency Enhancement Module (RPFBM) is as follows: Figure 2As shown, the input to this strategy is the initial feature map X∈F output from Stage 3, Stage 4, and Stage 5. C×W×H ;

[0059] First, perform k and l uniform slicing operations along the width and height dimensions on the feature map X to obtain N = k × l segmented feature maps X. i，j ∈F C×(W / k)×(H / l) ,i∈[1,k], j∈[1,l];

[0060] Then, a 1x1 convolution operation is used to explore the segmentation feature map X. i，j The importance of X is emphasized, and global average GAP pooling is used to calculate X. i，j Importance weighting factor b i，j :

[0061] b i，j =GAP(Relu(φ(X) i，j ))∈F 1×(W / k)×(H / l) )

[0062] Then, through a normalization operation, the weight factor b is... i，j The importance weight matrix B is reconstructed by dividing the matrix into blocks, where B(i,j)=b′. i，j

[0063]

[0064] Finally, the significant discriminative features are amplified by element-wise multiplication and summed with the initial feature map X to generate a significant feature map X. b :

[0065]

[0066] 5. Construction of the dual-path attention pooling strategy (BAP), specifically including the following steps:

[0067] Each region salient feature enhancement module (RPFBM) is followed by a bilinear attention pooling strategy (BAP), which uses multiple methods such as... Figure 3 The illustrated dual-line attention pooling (BAP) process explores local information hidden in subtle regions of the target object at multiple levels.

[0068] First, a salient feature map X is generated using 3x3 convolution. b Attention map A represents a specific part of the target object;

[0069] Then, the attention map A and the feature map X are multiplied element-wise using two tensors. b Multiplying them together yields the feature map F. k k = 1, 2, ..., M, and the feature extraction function GAP is used to extract the feature map F of the location.k The attentional features f of each specific region are obtained through processing. k ∈R 1×C ;

[0070] F k =A k ⊙X b (k = 1, 2, ..., M)

[0071] f k =GAP(F k )

[0072] Finally, the attention feature f k Stacked into a feature matrix P∈R M×C The feature matrix at different scales can hierarchically represent the salient features of an object.

[0073]

[0074] Location feature matrix P∈R M×C After passing through the fully linked layer, the fact prediction P is generated. T :

[0075] P r =C([Γ(A,X) b )]);

[0076] 6. Construction of the Counterfactual Attention Learning Network (CAL). Specific steps include:

[0077] Counterfactual attention learning network structure such as Figure 4 As shown, when you want to understand the effect of a variable, you can modify its value and observe the effect to analyze causal relationships. In the field of causal reasoning, this operation is called counterfactual intervention. This invention connects attention map A with feature map X. b The final prediction obtained after performing the BAP process is using a fully connected layer. During this process, the feature map X... b Together with attention map A, they determine the prediction result, satisfying a causal relationship structure. Drawing on the counterfactual intervention process described above, feature map X is preserved. b If the values ​​in attention map A remain unchanged, the effectiveness of attention map A can be evaluated by modifying them and observing the final prediction results.

[0078] Specifically, based on the size of attention map A, a fake attention map is generated. Fake attention map The values ​​of and are all random numbers and their sizes are exactly the same as those of attention map A, using a fake attention map. Counterfactual intervention can be performed, similar to attention map A, to counteract spurious attention maps. With feature map Xb The prediction result obtained using the fully connected layer after performing the BAP process is called the counterfactual prediction P. F :

[0079]

[0080] Using fact prediction P T and counterfactual prediction P F The difference between Y effect This can be used to measure the quality of effective information learned by attention map A, i.e., compared to incorrect attention maps. In comparison, attention map A improves the final prediction results.

[0081] Therefore, Y can be used effect As a supervisory signal, it explicitly guides the learning process of attention.

[0082] Y effect =P T -P F

[0083] Finally, for each branch, a fact prediction P is generated. 1T P 2T P 3T and counterfactual prediction P 1F P 2F P 3F The difference results Y of the corresponding branches effect Name them P1, P2, and P3, and sum them to generate the final prediction result P. C .

[0084] 7. Loss calculation, the specific steps include:

[0085] The preprocessed training dataset is fed into a multi-scale salient feature bilinear attention fine-grained classification network to obtain the predicted classification results, and the classification label y and the fact prediction P are evaluated separately. T and difference results Y effect Perform cross-entropy calculation, and sum the two to generate a new loss function L:

[0086] L = L ce (Y effect ,y)+L ce (P T ,y)

[0087] Calculate the loss of the classification result using the loss function.

[0088] 8. Network training, the specific steps of which include:

[0089] The Adam optimization algorithm was used to train a multi-scale salient feature bilinear attention fine-grained classification network to obtain the final model parameters of the multi-scale salient feature bilinear attention fine-grained classification network.

[0090] 9. Verification using comparative methods, specifically including:

[0091] As shown in Table 1, the comparative experimental results of multiple algorithms demonstrate that the method of this invention achieves an accuracy of 89.3% on the CUB-200-2011 (bird) dataset, significantly higher than Part-based R-CNNs, PoseNorm, KERL, and Mask-CNN with VGG as the backbone. Furthermore, it enables weakly supervised end-to-end training using only image-level labels. When using the same backbone network, ResNet50, and training with only label input, the method of this invention also significantly outperforms other algorithms under the same conditions, such as MAMC, CIN, NTS, API-Net, DTB-Net, LIO, and FDL. These weakly supervised algorithms often focus on the most salient features while neglecting potential subtle features, and their networks, lacking effective supervision signals, struggle to accurately locate part information. In contrast, the algorithm of this invention, through the application of RPFBM, MBAP, and CAL, incentivizes the network to autonomously explore potential discriminative information, accurately focusing on local region locations and improving the network's feature representation and model generalization capabilities. Therefore, the method of this invention exhibits superior performance.

[0092] Stanford Cars only provides category labels and bounding boxes, making it suitable for weakly supervised algorithms. Our method achieved 95.0% accuracy on this dataset, outperforming BCNN, MAMC, CIN, NTS, API-Net, DTB-Net, LIO, and FDL.

[0093] The Stanford Dogs dataset is more challenging than the two datasets mentioned above, with less variation between different categories and greater variation among samples within the same category. Only a few methods have been tested on this dataset. Our algorithm achieved 87.6% accuracy on this dataset, higher than MAMC and FDL, but slightly lower than the API-Net algorithm. However, API-Net has a complex training process, and as described in API-Net, achieving its experimental accuracy requires 240 images per batch, placing high demands on resources and demanding sophisticated equipment.

[0094] Table 1: Accuracy comparison of different algorithms on three datasets

[0095]

[0096]

[0097] 9. Ablation experiments and visualization analysis, the specific steps of which include:

[0098] The CUB-200-2011 (bird) dataset possesses a rich variety of species and balanced data. Therefore, this invention selected this dataset for ablation experiments to analyze the impact of each module on the final classification performance. The results are shown in Table 2. It can be seen that when using only the ResNet50 backbone network, the algorithm accuracy is 85.5%; introducing the RPFBM module improves the accuracy to 87.3%, a 1.8% increase; after adopting a bilinear attention pooling strategy, the algorithm accuracy improves to 88.7%; finally, utilizing counterfactual attention learning further enhances the classification accuracy, reaching 89.3%.

[0099] To gain a more intuitive understanding of the function of each module and to further verify the reliability of the method, this invention conducted visualization experiments, performing visual analysis on the feature maps of the backbone network ResNet50 and after introducing RPFBM, MBAP, and CAL. Figure 5 This is a visualization of the feature map of the present invention.

[0100] Depend on Figure 5 It can be seen that for the backbone network ResNet50, the model only focuses on some salient features while ignoring some subtle features. For example, it only focuses on the bird's head, ignoring the discriminative tail. After introducing the RPFBM module, the previously ignored subtle features are amplified, and the network begins to pay attention to more details besides the bird's head, but its focus still needs improvement. Figure 5 As shown in (d), after further introducing the MBAP module, the network can actively locate the magnified salient parts, and its attention to local features such as the bird's tail is significantly improved. However, there are localization errors in this process. In order to accurately locate distinguishable features, CAL is used to improve the network's ability to actively locate local features and correct the erroneous information of locating the background.

[0101] Table 2: Ablation experimental results of the algorithm of this invention on the CUB-200-2011 dataset

[0102]

[0103]

[0104] This invention also provides a multi-scale salient feature bilinear attention fine-grained object classification system, specifically including:

[0105] The network construction module is used to construct a multi-scale salient feature bilinear attention fine-grained classification network. The network includes a basic backbone network ResNet50. After Stage 3, Stage 4 and Stage 5 of the basic backbone network ResNet50, a Region Satisfactory Feature Enhancement Module (RPFBM) is connected. After the RPFBM, a bilinear attention pooling strategy (BAP) is connected. Both the RPFBM and the BAP are connected to the Counterfactual Attention Learning Network (CAL).

[0106] The loss function calculation module is used to establish the loss function using the classification label y, the output of the dual-line attention pooling strategy BAP, and the output of the dual-line attention pooling strategy BAP.

[0107] The model training module is used to train the multi-scale salient feature bilinear attention fine-grained classification network to obtain the multi-scale salient feature bilinear attention fine-grained classification network model.

[0108] The fine-grained classification module is used to feed the preprocessed training dataset into the multi-scale salient feature bilinear attention fine-grained classification network model to obtain fine-grained classification results.

[0109] Furthermore, the counterfactual attention learning network CAL incorporates a bilinear attention pooling strategy, BAP.

[0110] The present invention also provides a computer device, which includes a computer, a server or other terminal device with computing functions. The device includes a processor and a memory connected by a bus. The memory stores a program, and the program is executed by the processor. The program includes methods for performing the multi-scale salient feature bilinear attention fine-grained classification method as described above.

Claims

1. A multi-scale salient feature bilinear attention fine-grained object classification method, characterized in that, The specific steps are as follows: S1 constructs a multi-scale salient feature dual-line attention fine-grained classification network. The network includes a basic backbone network ResNet50. After Stage 3, Stage 4 and Stage 5 of the basic backbone network ResNet50, a Region Satisfactory Feature Enhancement Module (RPFBM) is connected. After the RPFBM, a dual-line attention pooling strategy (BAP) is connected. Both the RPFBM and the BAP are connected to the Counterfactual Attention Learning Network (CAL). The CAL contains the BAP. S2 uses category labels The loss function is constructed using the output of the bilinear attention pooling strategy BAP connected after the Region Saliency Enhancement Module RPFBM and the output of the bilinear attention pooling strategy BAP inserted into the Counterfactual Attention Learning Network CAL. S3 is used to train a multi-scale salient feature bilinear attention fine-grained classification network to obtain a multi-scale salient feature bilinear attention fine-grained classification network model. S4 feeds the preprocessed training dataset into the multi-scale salient feature bilinear attention fine-grained classification network model to obtain fine-grained classification results of objects; In S1, the construction of the Region Satisfactory Feature Enhancement Module (RPFBM) specifically includes: 1) Initial feature maps of the outputs of Stage 3, Stage 4, and Stage 5 of the basic backbone network ResNet50 Along the width and height dimensions k Shares and l A uniform slice was prepared to obtain N = k × l Each segmentation feature map , , ; 2) Employ 1x1 convolution operations and global averaging. GAP Pooling calculation to segment feature maps Weighting factors and weighting factors The importance weight matrix is ​​reconstructed by dividing the data into blocks. B , B ( i, j )= ; 3) Weight the importance matrix B and initial feature map Perform element-wise multiplication and then combine with the initial feature map X Sum the results to generate a salient feature map. .

2. The multi-scale salient feature bilinear attention fine-grained object classification method according to claim 1, characterized in that, In S1, the basic backbone network ResNet50 is used to obtain feature maps at different scales. ,in C , W , H The feature map represents the number of channels, width, and height, and consists of 5 stages, where stage 1 is used for preprocessing the input image, and stages 2 through 4 consist of residual blocks called Bottleneck.

3. The multi-scale salient feature bilinear attention fine-grained object classification method according to claim 1, characterized in that, In S1, the construction of the dual-path attention pooling strategy BAP specifically includes: 1) The salient feature map obtained by the Region Salient Feature Enhancement Module (RPFBM) Perform a 3x3 convolution operation to obtain attention map A; 2) Attention map A and feature map Perform element-wise tensor multiplication to obtain the local feature map. k=1,2,…,M, the feature extraction function GAP is used to extract the feature map of the location. The attentional characteristics of each specific region are obtained through processing. ; 3) Attention characteristics f k Stacked into feature matrices of different scales Partial feature matrix Fact predictions are generated after passing through a fully linked layer. : 。 4. The multi-scale salient feature bilinear attention fine-grained object classification method according to claim 1, characterized in that, In S1, the construction of the Counterfactual Attention Learning Network (CAL) specifically includes: 1) Generation and Attention Map A Fake attention maps of the same size Fake attention graph All values ​​in the data are random numbers; 2) Regarding spurious attention maps A dual-path attention pooling (BAP) strategy is applied, and counterfactual predictions are obtained after passing through a fully connected layer. : in, This is the salient feature map output by the Region Salient Feature Enhancement Module (RPFBM).

5. The multi-scale salient feature bilinear attention fine-grained object classification method according to claim 1, characterized in that, In S2, the output of the bilinear attention pooling strategy BAP connected after the Region Saliency Enhancement Module (RPFBM) is the fact prediction. The output of the bilinear attention pooling strategy BAP inserted into the counterfactual attention learning network CAL is the counterfactual prediction. Calculate fact prediction and counterfactual prediction Differences between Predicting facts respectively and difference results With category tags Perform cross-entropy calculation and sum the results to obtain the loss function. : 。 6. The multi-scale salient feature bilinear attention fine-grained object classification method according to claim 1, characterized in that, In S3, the Adam optimization algorithm is used to train a multi-scale salient feature bilinear attention fine-grained classification network to obtain the final multi-scale salient feature bilinear attention fine-grained classification network model.

7. A multi-scale salient feature bilinear attention fine-grained object classification system, characterized in that, Specifically, it includes: A network construction module is used to construct a multi-scale salient feature bilinear attention fine-grained classification network. The network includes a basic backbone network ResNet50. Each stage (Stage 3, Stage 4, and Stage 5) of the ResNet50 backbone network is connected to a Region Satisfactory Feature Enhancement Module (RPFBM). Each RPFBM is followed by a bilinear attention pooling strategy (BAP). Both the RPFBM and the BAP are connected to a Counterfactual Attention Learning Network (CAL). The CAL contains the BAP. The loss function calculation module is used to calculate the loss function using classification labels. The loss function is constructed using the output of the bilinear attention pooling strategy BAP connected after the Region Saliency Enhancement Module RPFBM and the output of the bilinear attention pooling strategy BAP inserted into the Counterfactual Attention Learning Network CAL. The model training module is used to train the multi-scale salient feature bilinear attention fine-grained classification network to obtain the multi-scale salient feature bilinear attention fine-grained classification network model. The fine-grained classification module is used to feed the preprocessed training dataset into the multi-scale salient feature bilinear attention fine-grained classification network model to obtain the fine-grained classification results of objects. The construction of the Region Satisfactory Feature Enhancement Module (RPFBM) specifically includes: 1) Initial feature maps of the outputs of Stage 3, Stage 4, and Stage 5 of the basic backbone network ResNet50 Along the width and height dimensions k Shares and l A uniform slice was prepared to obtain N = k × l Each segmentation feature map , , ; 2) Employ 1x1 convolution operations and global averaging. GAP Pooling calculation to segment feature maps Weighting factors and weighting factors The importance weight matrix is ​​reconstructed by dividing the data into blocks. B , B ( i, j )= ; 3) Weight the importance matrix B and initial feature map Perform element-wise multiplication and then combine with the initial feature map X Sum the results to generate a salient feature map. .

8. A computer device, characterized in that, The computer device includes a computer, server, or other terminal device with computing functions. The device includes a processor and a memory connected via a bus. The memory stores a program, which is executed by the processor. The program includes a method for executing the multi-scale salient feature bilinear attention fine-grained classification method according to any one of claims 1 to 6.

Images