A local diversity guided weakly supervised fine-grained image classification method and system

Abstract

The application provides a weakly supervised fine-grained image classification method and system guided by local diversity, constructs a weakly supervised fine-grained image classification network guided by local diversity, and the classification network comprises a basic backbone network, a cross-layer attention interaction module and a bilinear pooling module connected after Layer 3 and Layer 4 of the basic backbone network, and a random selection strategy is connected between the cross-layer attention interaction module and the bilinear pooling module; the weakly supervised fine-grained image classification network guided by local diversity is trained to obtain a weakly supervised fine-grained image classification network model guided by local diversity; and preprocessed training data sets are sent into the weakly supervised fine-grained image classification network model guided by local diversity to obtain an image classification result. The application solves the problem of inaccurate classification results caused by the problems that distinguishable features are too fine to be captured and local information is not effectively utilized in the existing fine-grained image classification task.

Description

A locally diversified guided weakly supervised fine-grained image classification method and system Technical Field

[0001] This invention belongs to the field of electronic information technology, specifically to a weakly supervised fine-grained image classification method and system guided by local diversification. Background Technology

[0002] Fine-grained image classification, also known as subclass image classification, aims to further subdivide images belonging to the same basic category (cars, dogs, flowers, birds, etc.). It has wide-ranging business needs and applications in industry and daily life. For example, it can be used for research on the identification of flora and fauna in the wild, providing an important technological foundation for the field of biology; it can be applied to visual tasks such as clothing inspection and recognition; it can be used for automated checkout services in retail scenarios; and it can be used for fast, accurate, and refined identification of high-speed vehicles. Due to the subtle inter-class differences and significant intra-class differences between subclasses, fine-grained image classification is more challenging than ordinary image classification tasks.

[0003] In image recognition algorithms, training can be divided into two main directions: strongly supervised and weakly supervised. Strong supervision involves using not only class labels but also bounding boxes and part annotations in the model. Weak supervision, on the other hand, trains the model using only class labels. For fine-grained tasks, the key to solving these problems is enabling the neural network to locate distinguishable parts and learn discernible features. Early solutions often employed strongly supervised methods. However, the practicality of strongly supervised algorithms is limited by their heavy reliance on manually labeled bounding boxes and part annotations, which are expensive to obtain. Thanks to the development of deep learning and related research, weakly supervised methods, which rely solely on class labels for classification without additional manual annotations, can achieve good classification performance. Weakly supervised classification algorithms not only reduce the cost of manual annotation but also better meet practical application needs, representing the current research trend.

[0004] Attention mechanisms can improve the focus on discriminative parts, thereby increasing the accuracy of classification tasks. While attention mechanisms can guide the model to focus on discriminative parts, their methods typically focus only on a few salient parts without fully exploring potential discriminative parts, and treat each feature in isolation, leading to inaccurate classification results when performing fine-grained classification of images in the same application scenario. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a locally diversified guided weakly supervised fine-grained image classification method and system. By constructing a locally diversified guided weakly supervised fine-grained classification network model, it solves the problem of inaccurate classification results caused by the difficulty in capturing distinguishable features that are too subtle and the lack of effective utilization of local information in existing fine-grained image classification tasks.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a locally diversified guided weakly supervised fine-grained image classification method, the specific steps of which are as follows:

[0007] S1 constructs a locally diversified guided weakly supervised fine-grained image classification network. The classification network includes a basic backbone network ConvNeXt, a cross-layer attention interaction module CAIM and a bilinear pooling module BP connected to Layer 3 and Layer 4 of the basic backbone network ConvNeXt, and a random selection strategy RSS between the cross-layer attention interaction module CAIM and the bilinear pooling module BP.

[0008] S2 trains a locally diversified guided weakly supervised fine-grained image classification network to obtain a locally diversified guided weakly supervised fine-grained image classification network model.

[0009] S3 feeds the preprocessed training dataset into a locally diversified guided weakly supervised fine-grained image classification network model to obtain image classification results.

[0010] Furthermore, the basic backbone network ConvNeXt consists of four layers: Layer 1 is used for preprocessing the input image; Layers 2-4 are each composed of multiple ConvNeXt blocks; and Layers 3 and 4 of the basic backbone network ConvNeXt output initial feature maps D∈R at different scales. C×W×H Where C, W, and H represent the number of channels, width, and height of the feature map.

[0011] Furthermore, the construction of the cross-layer attention interaction module CAIM involves the following steps:

[0012] 1) Initial feature maps D∈R at different scales obtained from the backbone network Layer3 and Layer4 C×W×H Perform 3x3 convolutions to obtain multi-level attention maps

[0013] 2) Multi-level attention maps Spatial interaction and channel interaction processing are performed to obtain feature map A, which completes the spatial interaction. s12 A s21 Feature map A that completes channel interaction c12 A c21;

[0014] 3) A, which has completed spatial interaction, will be respectively... s12 A interacting with the channel c12 Combining these elements yields a diverse feature map A. 12 A will complete spatial interaction s21 A interacting with the channel c21 Combining these elements yields a diverse feature map A. 21 .

[0015] Furthermore, the spatial interaction includes:

[0016] 1) Multi-level attention maps The feature map is obtained by performing size transformation. L1 = W1 × H1, L2 = W2 × H2;

[0017] 2) Perform an inner product operation on feature maps A′1 and A′2 to obtain the spatial similarity matrix W1. After normalizing -W1, the spatial interaction feature map M1 is obtained.

[0018]

[0019] 3) Map the spatial interaction feature map M1 to feature maps A′1 and A′2 to obtain feature map A′. s12 With A′ s21 :

[0020] A′ s12 =M1 T ×A′1+A′ T 2

[0021] A′ s21 =M1×A ′T 2+A1

[0022] 4) For feature map A′ s12 With A′ s21 Size transformation yields feature maps that reflect spatial dependencies. and

[0023] Furthermore, the channel interaction includes:

[0024] 1) Downsampling and size transformation are performed on the multi-level attention map A2 to obtain the feature map. The feature map A′1 is obtained by resizing the multi-level attention map A1. ′T 1 and A′ 2d As a diagram;

[0025] 2) For A ′T 1 and A′2d The channel similarity matrix W2 obtained by performing inner product operation is used to evaluate the similarity of image pairs, and -W2 is then used to calculate the similarity. T After normalization, the channel interaction feature map M2 is obtained:

[0026] M2 = softmax(-W2) T )∈[0,1] C′×C′ W2=A′ T 1×A′ 2d

[0027] 3) Map the channel interaction feature map M2 to feature maps A′1 and A′2. 2d The feature map A′ is obtained. c12 With A′ c21 :

[0028] A′ c12 =M2×A ′T 1+A ′T 2d

[0029]

[0030] 4) For feature map A′ c12 Dimension conversion yields A c12 For feature map A c21 A was obtained by performing a size conversion. c21 .

[0031] Furthermore, the construction of the random selection strategy RSS involves the following specific steps:

[0032] 1) Feature map F∈R of the cross-layer attention interaction module CAIM output along the width dimension. C′×W×H Divide the slice into n equal parts to obtain n feature slices F (k) ∈R C′×(W / n)×H k∈[1,n];

[0033] 2) For each feature slice F (k) The most significant suppression branch, feature enhancement branch, or no-operation branch is randomly selected for data processing to obtain S. (k) :

[0034]

[0035] Where α, β, and γ represent the probabilities of selecting the most significant suppression branch, the feature enhancement branch, or the no-operation branch, respectively, and α+β+γ=1;

[0036] 3) Process each feature slice F(k) The aggregated attention map A∈R is obtained by concatenating the segments according to their width dimensions. C′×W×H :

[0037] A = concat(S) (k) ).

[0038] Furthermore, the most significant inhibition branch includes:

[0039] 1) For feature slice F (k) Perform channel average pooling operation to obtain F p(k) ∈R (W / n)×H :

[0040]

[0041] 2) Based on feature slice F (k) The maximum intensity pixel value is used to set the threshold rate δ to generate the demask P. (k)drop Pixels larger than the threshold are set to 0, while pixels smaller than the threshold are set to 1.

[0042]

[0043] 3) Remove mask P (k)drop Apply dot product to feature slice F (k) Obtain the eliminated feature map;

[0044] The feature enhancement branch includes:

[0045] 1) For feature slice F (k) The sigmoid activation function is used to generate the enhanced mask P. (k)important :

[0046] P (k)important =sigmoid(F P(k) )∈[0,1] (W / n)×H

[0047] 2) Enhance the mask P (k)important Apply dot product to slice F (k) Obtain the enhanced feature map;

[0048] The no-operation branch pair feature slice F (k) No action will be taken.

[0049] Furthermore, bilinear pooling (BP) is applied to the outputs of the random selection strategy RSS and the basic backbone network ConvNeXt. The specific steps include:

[0050] 1) Obtain the initial feature map D output by the basic backbone network ConvNeXt and the aggregated attention map A∈R from the output of the random selection policy RSS. C′×H×W ;

[0051] 2) Multiply the convergent attention map A with the initial feature map D using element-wise dot product to obtain the part feature map D. k ;

[0052] D k =A k ⊙D(k=1,2,...,C′)

[0053] 3) Use Global Max Pooling (GMP) to analyze the feature map D. k Processing is performed to obtain attention features of each specific region d ⊙ ∈R 1×C :

[0054] d k =GMP(D k )

[0055] 4) Incorporate attention features d k Stacked into a feature matrix P∈R M×C :

[0056] P = (d1, d2, ..., d M ) T .

[0057] Furthermore, Layer 3 and Layer 4 of the basic backbone network ConvNeXt each yield a local feature matrix P∈R. M×C The feature maps y are obtained by concatenating the feature matrices of these two parts in terms of dimensions. The feature maps y are then mapped to one-dimensional feature vectors. Finally, the image is classified by softmax logistic regression using a fully connected layer.

[0058] This invention also provides a locally diversified guided weakly supervised fine-grained image classification system, comprising:

[0059] The classification network construction module is used to construct a weakly supervised fine-grained image classification network guided by local diversity. The classification network includes a basic backbone network ConvNeXt, a cross-layer attention interaction module CAIM and a bilinear pooling module BP connected to Layer 3 and Layer 4 of the basic backbone network ConvNeXt, and a random selection strategy RSS connected between the cross-layer attention interaction module CAIM and the bilinear pooling module BP.

[0060] The classification network training module is used to train a locally diversified guided weakly supervised fine-grained image classification network, resulting in a locally diversified guided weakly supervised fine-grained image classification network model.

[0061] The image classification module is used to feed the preprocessed training dataset into a locally diversified guided weakly supervised fine-grained image classification network model to obtain the image classification results.

[0062] Compared with the prior art, the present invention has at least the following beneficial effects:

[0063] This invention proposes a weakly supervised fine-grained image classification method guided by local diversity. First, ConvNeXt is used as the backbone network to extract multi-level initial fine-grained features to obtain feature maps. Then, a cross-layer attention interaction module (CAIM) is designed, using different levels of attention maps to represent different parts of the object. Multi-level attention maps are established, and channel interaction and spatial interaction are used to improve the semantic expression of the attention maps and share the mined information. Next, to avoid feature convergence, a random selection strategy (RSS) is proposed, and the random selection method promotes the attention maps to capture more discriminative local information, further promoting the network to obtain richer local features. Finally, the attention maps and feature maps are fused using a bilinear pooling module to construct a powerful feature representation to enhance the network's fitting ability. The classification task is completed through a fully connected layer, which solves the problems of overly subtle distinguishable features that are difficult to capture and the lack of effective utilization of local information in fine-grained image classification tasks, and improves the classification accuracy of images belonging to the same basic category. Attached Figure Description

[0064] Figure 1. Structure diagram of a fine-grained image classification network guided by local diversification;

[0065] Figure 2. Structure diagram of the cross-layer attention interaction module;

[0066] Figure 3. Structure diagram of the random selection strategy;

[0067] Figure 4 shows the visualization effect. Detailed Implementation

[0068] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0069] The process of the locally diversified guided weakly supervised fine-grained image classification method of the present invention is as follows:

[0070] 1. Download images from the CUB-200-2011 (birds 1), NABirds (birds 2), ISIA Food-200 (food), and ancient pagoda / building datasets. Screen the images to ensure their integrity and preprocess them. During preprocessing, this invention employs data augmentation methods to randomly crop, rotate, and scale the sample images, increasing the number of samples in the dataset and enhancing the robustness of the CNN model.

[0071] 2. Construct a weakly supervised fine-grained image classification network guided by local diversity. Specific steps include:

[0072] The convolutional neural network structure based on local diversification guidance, as shown in Figure 1, includes: a basic backbone network (ConvNeXt), a cross-layer attention interaction module (CAIM), a random selection strategy (RSS), a bilinear pooling module (BP), and a cross-layer concatenation classification operation. The CAIM module models the feature interactions between the outputs of Layer 3 and Layer 4 of the backbone network. After passing through CAIM, it outputs diversified feature maps. The RSS strategy is then applied to these diversified feature maps to output local feature matrices. Finally, the two local feature matrices are concatenated and input into the classifier to complete the classification task.

[0073] First, a basic backbone network (ConvNeXt) is constructed, which mainly consists of four layers: Layer 1 has a simple structure and can be regarded as a preprocessing of the input image; Layers 2-4 are all composed of multiple ConvNeXt blocks and have similar structures. The input image passes through different layers to obtain initial feature maps of different scales, denoted as D∈R. C×W×H Where C, W, and H represent the number of channels, width, and height of the feature map.

[0074] Secondly, for fine-grained tasks, capturing more discriminative local information is crucial to classification performance. Therefore, this invention generates multi-level attention maps from the backbone networks Layer 3 and Layer 4 using 3x3 convolutions. Two multi-level attention maps can represent specific parts of the target object. Compared with other attention mechanism models, this method can more easily locate most object parts, thus obtaining better local feature representations. In the attention map, the number of channels C′ is set to 16 based on engineering experience in this invention.

[0075] At the same time, if the feature representation of the object component is directly output after obtaining it, it will limit the model's ability to compare clues from different features. Therefore, these different global features should not be treated in isolation. A more reasonable approach is to perform cross-layer interaction modeling on attention maps at different levels, forcing the network to share information mined from different layers and enhancing semantic complementarity.

[0076] Furthermore, in the aforementioned cross-layer attention interaction module, without constraints, attention maps can converge, meaning they easily focus on salient regions of the same target object, neglecting other less salient regions with similar discriminative power. Therefore, this invention proposes a random selection strategy, designing three candidate branches: the most salient suppression branch, the feature enhancement branch, and the no-operation branch. The most salient suppression branch suppresses the most discriminative parts, the feature enhancement branch rewards the most discriminative parts, and the no-operation branch, as the name suggests, takes no action. The three candidate branches, executed randomly, ensure a balance between punishment and reward while not neglecting original features, forcing the attention map to focus on more effective local information with the highest cost-effectiveness.

[0077] After the random selection strategy is implemented, each channel in the attention map can represent features from different parts of the image. If it can be effectively fused with the original feature map, a more powerful feature representation can be constructed. Therefore, this invention utilizes the bilinear pooling module (BP) to aggregate the attention map A output by the random selection strategy and the initial feature map D.

[0078] In the classifier part, this invention maps the fused and stitched feature map into a one-dimensional feature vector, and uses a fully connected layer to achieve the final classification of the image through softmax logistic regression.

[0079] 3. The execution of the basic backbone network includes the following specific steps:

[0080] Images from the preprocessed training dataset are fed into ConvNeXt to generate initial feature maps at different scales. These initial feature maps are labeled as D∈R. C×W×H Where C, W, and H represent the number of channels, width, and height of the feature map.

[0081] 4. Construction of the cross-layer attention interaction module CAIM.

[0082] Figure 2 is a structural diagram of the cross-layer attention interaction module of the present invention. As shown in Figure 2, the cross-layer attention interaction module is mainly divided into channel interaction and spatial interaction modeling.

[0083] In the spatial interaction part, firstly, the initial feature maps D∈R obtained from the backbone network Layer3 and Layer4 at different scales are processed. C×W×H Multi-level attention maps generated using 3x3 convolution. Information is reorganized from image pairs and resized. L1 = W1 × H1, L2 = W2 × H2;

[0084] Next, image pair similarity is evaluated. Specifically, the spatial similarity matrix W1 is obtained by performing an inner product operation on A′1 and A′2, and the matrix elements... Represents the i-th term of A′1 th The j-th pixel and A′2 th The similarity between two pixels is calculated by dividing them into 10 pixels. The lower the similarity between two pixels, the stronger the complementarity between them. Therefore, the -W1 normalization operation is used to express the interaction relationship between image pairs, resulting in the spatial interaction feature map M1:

[0085]

[0086] Then, the spatial interaction feature map M1 is mapped to A′1 and A′2 to obtain A′. s12 With A′ s21 :

[0087] A′ s12 =M1 T ×A′1+A ′T 2

[0088] A′ s21 =M1×A ′T 2+A1

[0089] Finally, regarding A′ s12 With A′ s21 Size transformation yields feature maps that reflect spatial dependencies. and This allows us to learn more distinguishable features and output feature maps. and

[0090] In the channel interaction part, each channel in the attention map can be regarded as a feature representation of a specific part. If a correlation model is established between different channels, the complementary information between channels can be enhanced. Based on this:

[0091] First, construct image pairs A′1 and A′. 2d , where feature map It is obtained by downsampling and size transformation of the multi-level attention map A2; feature map A′1 is the same as above, obtained by size transformation of the multi-level attention map A1;

[0092] Secondly, very similar to the spatial interaction part mentioned above, through the feature map A ′T 1 and A′ 2d The channel similarity matrix W2 obtained by performing inner product operation is used to evaluate the similarity of image pairs, and -W2 is then applied. T After normalization, the interaction relationships between image pairs are expressed, resulting in the channel interaction feature map M2:

[0093] M2 = softmax(-W2) T )∈[0,1]C′×C′ W2 = A ′T 1×A′ 2d

[0094] Unlike the spatial interaction section, the channel interaction section mainly explores complementary information along the channel dimension;

[0095] Then, the channel interaction feature map M2 is mapped to feature maps A′1 and A′. 2d The feature map A′ is obtained. c12 With A′ c21 :

[0096] A′ c12 =M2×A ′T 1+A ′T 2d

[0097]

[0098] Finally, the feature map The feature map is obtained by performing size transformation. For feature maps This requires an additional step: after the initial processing step, the dimensions are converted to L1×C′, and then a further dimension conversion is performed to obtain the feature map.

[0099] After completing the above operations, A, which has completed spatial interaction, will be... s12 A interacting with the channel c12 The combination yields a rich and diverse feature map A. 12 A will complete spatial interaction s21 A interacting with the channel c21 The combination yields a rich and diverse feature map A. 21 .

[0100] 5. Construction of a random selection strategy RSS feed. Specific steps include:

[0101] Figure 3 is a structural diagram of the random selection strategy of the present invention. Theoretically, the input of this strategy can be any feature map F. In this section, F specifically refers to the diverse feature map A output by the aforementioned cross-layer attention interaction module. 12 and A 21 .

[0102] First, along the width dimension, the feature map F∈R C′×W×H Performing an n-part slicing operation yields n feature slices F. (k) ∈R C′×(W / n)×H k∈[1,n];

[0103] Then, each feature slice F (k)The corresponding operation is performed by randomly selecting a candidate branch. n is determined by engineering experience, and is set to 7 in this invention.

[0104] 1) For the most significant suppression branch, for feature slice F (k) Perform channel average pooling operation to obtain F p(k) ∈R (W / n)×H .

[0105]

[0106] Feature slice F (k) The value range of each pixel is the same as that of the input feature map, representing the key feature expression obtained by the classification model. Since the randomly selected global diversity classification network proposed in this invention is trained for a classification task, the feature slice F... (k) It can approximately reflect the spatial distribution of the most discriminative parts; the higher the value of its element, the stronger the discriminative power. That is, for classification tasks, feature slice F (k) The intensity of each pixel in the image represents its discriminative power. To eliminate the most discriminative parts, the intensity is determined based on the feature slice F. (k) The maximum intensity pixel value is used to set the threshold rate δ to generate the demask P. (k)drop Pixels larger than the threshold are set to 0, while pixels smaller than the threshold are set to 1.

[0107]

[0108] Finally, the mask P will be eliminated. (k)drop Apply dot product to slice F (k) The eliminated feature map is obtained.

[0109] 2) For the feature enhancement branch, for feature slice F (k) The sigmoid activation function is used to generate the enhanced mask P. (k)important :

[0110] P (k)important =sigmoid(F P(k) )∈[0,1] (W / n)×H

[0111]

[0112] 3) For branches without operations, then for feature slice F (k) No action will be taken.

[0113] The probabilities of the three candidate branches are denoted by α, β, and γ, respectively, where α + β + γ = 1. Each feature slice F...(k) The selection feature S is obtained by randomly selecting one of the branches. (k) .

[0114]

[0115] Finally, each selected feature S after processing is... (k) Concatenate the segments according to their width dimension to obtain the aggregated attention map A∈R. c′×W×H :

[0116] A = concat(S) (k) ).

[0117] 6. Bilinear pooling (BP) module, the specific steps include:

[0118] After implementing the random selection strategy, the aggregated attention graph A∈R is formed. C′×H×W Each channel in the image can represent features from different parts of the image. If these channels can be effectively fused with the original feature map D, a more powerful feature representation can be constructed. Therefore, this invention utilizes the bilinear pooling module (BP) to aggregate the attention map A and the initial feature map D. The specific process is as follows:

[0119] First, the convergent attention map A is multiplied by the initial feature map D using element-wise dot product to obtain the part feature map D. k .

[0120]

[0121] Then, global max pooling (GMP) is used to analyze the local feature map D. k Processing is performed to obtain attention features of each specific region d k ∈R 1×C .

[0122] d k =GMP(D k )

[0123] Finally, the attention feature d k Stacked into a feature matrix P∈R M×C .

[0124] P = (d1, d2, ..., d M ) T

[0125] 7. Feature splicing, the specific steps include:

[0126] After passing through CAIM and RSS, Layer 3 and Layer 4 of the basic backbone network ConvNeXt will yield two different feature matrices. These two feature matrices are then concatenated dimensionally to obtain the feature map y, which is then input into the classifier for classification.

[0127] In the classifier part, this invention maps the fused and stitched feature map y into a one-dimensional feature vector, and uses a fully connected layer to achieve the final classification of the image through softmax logistic regression.

[0128] 8. Loss calculation, the specific steps include:

[0129] The preprocessed training dataset is fed into a locally diversified guided weakly supervised fine-grained image classification network to obtain the predicted classification results. The loss value of the classification results is calculated using the loss function and Euclidean distance. The Adam optimization algorithm is then used to train the locally diversified guided weakly supervised fine-grained image classification network to obtain the final locally diversified guided weakly supervised fine-grained image classification network model.

[0130] 9. Visual analysis, the specific steps of which include:

[0131] To further verify the effectiveness of the algorithm of this invention, a visualization experiment will be conducted. Figure 4 shows the visualization effect of this invention on the CUB-200-2011 dataset and the self-built Guta dataset. Compared with the original baseline model, the method of this invention can not only focus on the most salient region, but also pay attention to other neglected key points. This is due to the combined effect of the cross-layer attention interaction module CAIM and the random selection strategy RSS, which forces the network to fully explore multiple different discriminative features.

[0132] 10. Verification using comparison methods

[0133] This invention employs a superior feature extraction backbone network to construct a powerful initial feature representation, effectively improving algorithm accuracy. It establishes a feature interaction method to strengthen the association of multi-level attention maps and enhance the ability to mine semantic complementary information. A random selection strategy is proposed, designing three operations: "most significant suppression," "feature enhancement," and "no operation." By randomly executing these operations, the attention map is forced to learn more comprehensive local information, increasing the network's focus on more local features. Extensive experiments have been conducted on multiple datasets to demonstrate the effectiveness of the proposed method.

[0134] This invention will conduct comparative experiments on the CUB-200-2011 dataset. Image resolution and backbone network are important factors affecting the classification accuracy of fine-grained algorithms. Therefore, Table 1 lists the input image size and backbone network used in the comparative methods. The experimental results in Table 1 show that the method of this invention achieves an excellent performance of 92.3% accuracy on this dataset, which is far superior to Part-based R-CNNs, PoseNorm, KERL, Mask-CNN, and BCNN. FBSD and API-Net lead other CNN-based classification algorithms with performance of 89.8% and 90.0%, respectively, but both lag behind the method of this invention. Transformer-based methods such as RAMS-Trans, TPSKG, AFTrans, FFVT, and TransFG have achieved relatively good results, while the method of this invention, using ConvNeXt as the backbone network combined with a cross-layer attention interaction module and a random selection strategy, achieves even better performance.

[0135] Table 1: Accuracy comparison of different algorithms on the CUB-200-2011 (Birds 1) dataset

[0136]

[0137]

[0138] NABirds contains 48,562 bird images across 555 categories, with approximately 100 images per category. It is a larger, more fine-grained dataset than CUB-200-2011, making it more challenging for fine-grained tasks. Table 2 shows that the method described in this invention achieved an accuracy of 91.5% on this dataset, outperforming PC-CNN, MaxEnt, Cross-X, HGNet, and GaRD; and outperforming the relatively excellent TPSKG and TransFG by 1.4% and 0.7%, respectively. This demonstrates that the method described in this invention maintains good performance even on large datasets.

[0139] Table 2: Accuracy comparison of different algorithms on the NABirds dataset

[0140]

[0141] The ISIA Food-200 dataset contains 200 food categories, approximately 200,000 food images, and 319 ingredients. Compared to the two datasets mentioned above, this dataset has higher diversity and a larger data volume; therefore, only a few methods have attempted experiments on this dataset. As shown in Table 3, IG-CMAN and TPSKG achieved accuracies of 67.5% and 69.5%, respectively, on this dataset. The method of this invention, however, achieved 72.8% diversity of multi-scale local features on this dataset, which is 5.3% and 3.3% higher than IG-CMAN and TPSKG, respectively.

[0142] Table 3: Accuracy Comparison of Different Algorithms on the ISIA Food-200 Dataset

[0143]

[0144]

[0145] For the Guta dataset, this invention collected 13,815 Guta images, encompassing 229 categories, of which 6,915 images were used for training and 6,900 for validation. On this dataset, experiments were first conducted on several backbone networks commonly used for fine-grained tasks. As shown in Table 4, ResNet50, ResNet101, DenseNet161, and ViT achieved accuracies of 92.3%, 93.4%, 93.6%, and 93.8% respectively on the Guta dataset. Based on the experiments on the above three datasets, this invention selected several high-performing algorithms for comparative experiments on this dataset, specifically including FBSD, PMG, FFVT, and TransFG. The results are shown in Table 4. The method of this invention achieved an accuracy of 95.2% on the self-built Guta dataset, outperforming other algorithms.

[0146] Table 4: Accuracy Comparison of Different Algorithms on the Guta dataset

[0147]

[0148] This invention also provides a locally diversified guided weakly supervised fine-grained image classification system, comprising:

[0149] The classification network construction module is used to construct a weakly supervised fine-grained image classification network guided by local diversity. The classification network includes a basic backbone network ConvNeXt, a cross-layer attention interaction module CAIM and a bilinear pooling module BP connected to Layer 3 and Layer 4 of the basic backbone network ConvNeXt, and a random selection strategy RSS connected between the cross-layer attention interaction module CAIM and the bilinear pooling module BP.

[0150] The classification network training module is used to train a locally diversified guided weakly supervised fine-grained image classification network, resulting in a locally diversified guided weakly supervised fine-grained image classification network model.

[0151] The image classification module is used to feed the preprocessed training dataset into a locally diversified guided weakly supervised fine-grained image classification network model to obtain the image classification results.

Claims

1. A locally diversified guided weakly supervised fine-grained image classification method, characterized in that, The specific steps are as follows: S1 Construct a locally diversified guided weakly supervised fine-grained image classification network. The classification network includes a basic backbone network ConvNeXt, a cross-layer attention interaction module CAIM and a bilinear pooling module BP connected to Layer 3 and Layer 4 of the basic backbone network ConvNeXt. A random selection strategy RSS connects the cross-layer attention interaction module CAIM and the bilinear pooling module BP. S2 Train the locally diversified guided weakly supervised fine-grained image classification network to obtain the locally diversified guided weakly supervised fine-grained image classification network model. S3 Feed the preprocessed training dataset into the locally diversified guided weakly supervised fine-grained image classification network model to obtain the image classification results. The basic backbone network ConvNeXt includes 4 layers. Layer 1 is used for preprocessing the input image. Layers 2-4 are all composed of multiple ConvNeXt Blocks. Layers 3 and 4 of the basic backbone network ConvNeXt output initial feature maps at different scales. Where C, W, and H represent the number of channels, width, and height of the feature map; the cross-layer attention interaction module CAIM is constructed, and the specific steps are as follows: 1) Initial feature maps of different scales obtained from Layer 3 and Layer 4 of the backbone network are processed. Perform 3x3 convolutions to obtain multi-level attention maps 、 ;2) Multi-level attention maps 、 Spatial interaction and channel interaction processing are performed to obtain the feature map of the completed spatial interaction. 、 Feature maps that complete channel interactions 、 ;3) Each completed spatial interaction Interacting with channels Combining these elements yields a diverse feature map A. 12 This will complete spatial interaction Interacting with the channel Combining these elements yields a diverse feature map A. 21 The construction of the random selection strategy RSS includes the following steps: 1) Analyzing the feature map output by the cross-layer attention interaction module CAIM along the width dimension. Divide the slice into n equal parts to obtain n feature slices. ;2) For each feature slice The most significant suppression branch, feature enhancement branch, or no-operation branch are randomly selected for data processing to obtain selected features. ： in, These represent the probabilities of selecting the most significant suppression branch, the feature enhancement branch, or the no-operation branch, respectively. ;3) Process each selected feature The aggregated attention map is obtained by concatenating the segments along their width dimension. ： 。 2. The locally diversified guided weakly supervised fine-grained image classification method according to claim 1, characterized in that, The spatial interaction includes: 1) attention maps at multiple levels 、 The feature map is obtained by performing size transformation. 、 ， ， ;2) For feature maps and Perform inner product operation to obtain spatial similarity matrix W1, then... The spatial interaction feature map is obtained after normalization. ： 3) Spatial interaction feature map Mapping to feature map 、 get and ： 4) Feature map and Size transformation yields feature maps that reflect spatial dependencies. and 。 3. The locally diversified guided weakly supervised fine-grained image classification method according to claim 1, characterized in that, The channel interaction includes: 1) downsampling and resizing the multi-level attention map A2 to obtain a feature map. Feature maps are obtained by scaling the multi-level attention map A1. ,Will As a graph relative; 2) for The channel similarity matrix W2, obtained by performing inner product operations, is used to evaluate the similarity of image pairs, and then... The channel interaction feature map is obtained after normalization. ： 3) Channel interaction feature map Mapping to feature map 、 Obtain feature map and ： 4) Feature map Dimension conversion is performed to obtain For feature maps The above was obtained through size conversion. 。 4. The locally diversified guided weakly supervised fine-grained image classification method according to claim 1, characterized in that, The most significant suppression branch includes: 1) for feature slices Perform channel average pooling operation to obtain ： 2) Based on feature slices Set the threshold rate for the pixel value with the highest intensity. Generate elimination mask Pixels larger than the threshold are set to 0, while pixels smaller than the threshold are set to 1. 3) Remove mask Apply dot product to feature slices Obtain the eliminated feature map; the feature enhancement branch includes: 1) processing feature slices Use the sigmoid activation function to generate the enhanced mask. ： 2) Enhance the mask Applying dot product to slices The enhanced feature map is obtained; the no-operation branch pairs feature slices. No action will be taken.

5. The locally diversified guided weakly supervised fine-grained image classification method according to claim 1, characterized in that, Bilinear pooling (BP) is applied to the outputs of the random selection policy RSS and the basic backbone network ConvNeXt. The specific steps include: 1) Obtaining the initial feature map D of the output of the basic backbone network ConvNeXt and the aggregated attention map of the output of the random selection policy RSS. 2) Multiply the convergent attention map A with the initial feature map D using element-wise dot product to obtain the part feature map. ； 3) Use Global Max Pooling (GMP) to analyze the feature maps of different parts. Processing is performed to obtain attentional characteristics of specific areas. ： 4) Attention feature d k Stacked into a feature matrix ： 。 6. The locally diversified guided weakly supervised fine-grained image classification method according to claim 5, characterized in that, Layer 3 and Layer 4 of the basic backbone network ConvNeXt each yield a local feature matrix. The feature maps y are obtained by concatenating the feature matrices of these two parts in terms of dimensions. The feature maps y are then mapped to one-dimensional feature vectors. Finally, the image is classified by softmax logistic regression using a fully connected layer.

7. A locally diversified guided weakly supervised fine-grained image classification system, characterized in that, include: The classification network construction module is used to construct a weakly supervised fine-grained image classification network guided by local diversity. The classification network includes a basic backbone network ConvNeXt, a cross-layer attention interaction module CAIM and a bilinear pooling module BP connected to Layer 3 and Layer 4 of the basic backbone network ConvNeXt, and a random selection strategy RSS between the cross-layer attention interaction module CAIM and the bilinear pooling module BP. The classification network training module trains a locally diversified guided weakly supervised fine-grained image classification network, resulting in a locally diversified guided weakly supervised fine-grained image classification network model. The image classification module feeds the preprocessed training dataset into the locally diversified guided weakly supervised fine-grained image classification network model to obtain image classification results. The basic backbone network ConvNeXt consists of four layers: Layer 1 is used for preprocessing the input image; Layers 2-4 are each composed of multiple ConvNeXt blocks; and Layers 3 and 4 of the basic backbone network ConvNeXt output initial feature maps at different scales. Where C, W, and H represent the number of channels, width, and height of the feature map; the cross-layer attention interaction module CAIM is constructed, and the specific steps are as follows: 1) Initial feature maps of different scales obtained from Layer 3 and Layer 4 of the backbone network are processed. Perform 3x3 convolutions to obtain multi-level attention maps 、 ;2) Multi-level attention maps 、 Spatial interaction and channel interaction processing are performed to obtain the feature map of the completed spatial interaction. 、 Feature maps that complete channel interactions 、 ;3) Each completed spatial interaction Interacting with channels Combining these elements yields a diverse feature map A. 12 This will complete spatial interaction Interacting with the channel Combining these elements yields a diverse feature map A. 21 The construction of the random selection strategy RSS includes the following steps: 1) Analyzing the feature map output by the cross-layer attention interaction module CAIM along the width dimension. Divide the slice into n equal parts to obtain n feature slices. ;2) For each feature slice The most significant suppression branch, feature enhancement branch, or no-operation branch are randomly selected for data processing to obtain selected features. ： in, These represent the probabilities of selecting the most significant suppression branch, the feature enhancement branch, or the no-operation branch, respectively. ;3) Process each selected feature The aggregated attention map is obtained by concatenating the segments along their width dimension. ： 。

Images