Multi-granularity image classification methods, devices, electronic equipment and storage media

Abstract

This application discloses a multi-granularity image classification method, apparatus, electronic device, and storage medium. The classification method is implemented through a hierarchical multi-granularity image classification model. The method includes: acquiring feature maps with multi-scale information; extracting and fusing feature vectors of different granularities to obtain multi-scale fused feature vectors at each level; performing hierarchical feature mapping on the multi-scale fused feature vectors at each level; and fusing the mapped feature vectors with the mapped fine-grained feature vectors. The loss function of the hierarchical multi-granularity image classification model includes a triplet loss based on class centers. The method in this application increases the local detail information of the network, fuses multi-granular features at each class level, solves the problem of different regions corresponding to coarse and fine granularities between levels, and enhances the feature representation of the image and improves the classification accuracy of images at each level of granularity by using a triplet loss based on class centers.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a multi-granularity image classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] Fine-grained image classification (FGIC) aims to distinguish fine-grained image categories under the same coarse-grained labels, such as the "species" of a bird, the "brand" of a car, and the "model" of an airplane. FGIC is gaining increasing attention from industry and academia and has already seen some concrete applications, such as automated biodiversity monitoring, smart retail, and intelligent transportation.

[0003] The FGIC method in related technologies typically utilizes attention mechanisms or designs specific loss functions to optimize feature representations, relying on discrete label information and failing to fully leverage the similarity relationships between labels to train the network. This single-label learning approach lacks a guided learning process and suffers from insufficient information.

[0004] Related technologies employ the hierarchical multi-granularity classification (HMC) task, which enhances the feature representation capability of fine-grained objects by using hierarchical multi-granularity labels. The HMC task aims to embed a hierarchical label structure into a loss function or network architecture using hierarchical multi-granularity labels to classify images for each level. These methods are based on high-level semantic information for hierarchical multi-granularity classification, rarely considering the contribution of intermediate semantic features with more detailed information to multi-granularity classification. Furthermore, the labels between layers are independent of each other, and there is no interaction between the semantic information of different layers. Summary of the Invention

[0005] The purpose of this application is to provide a multi-granularity image classification method, apparatus, electronic device, and storage medium. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0006] According to one aspect of the embodiments of this application, a multi-granularity image classification method is provided, implemented through a hierarchical multi-granularity image classification model, the classification method comprising:

[0007] Obtain feature maps with multi-scale information;

[0008] Extract and fuse feature vectors at different levels of granularity to obtain multi-scale fused feature vectors at each level;

[0009] Hierarchical feature mapping is performed on the multi-scale fused feature vectors of each level;

[0010] The mapped feature vector is fused with the mapped fine-grained feature vector.

[0011] The loss function of the hierarchical multi-granularity image classification model includes a triplet loss based on class center.

[0012] In some embodiments of this application, the hierarchical multi-granularity image classification model uses a ResNet-50 pre-trained on ImageNet-1K as the backbone network; the feature map for obtaining multi-scale information includes:

[0013] The feature map F obtained from the third block of ResNet-50 m ∈R C×2W×2H As intermediate semantic information; the feature map F obtained from the 4th block of ResNet-50 is taken. h ∈R C×W×H As high-level semantic information, C, W, and H represent the number of channels, width, and height of the fourth feature map of the backbone network ResNet-50, respectively.

[0014] In some embodiments of this application, the step of extracting and fusing feature vectors at different levels of granularity to obtain multi-scale fused feature vectors at various levels includes:

[0015] Granular information is extracted from the high-level semantic feature map to obtain the high-level semantic feature vector;

[0016] Granular information is extracted from the intermediate semantic feature map to obtain the intermediate semantic feature vector;

[0017] The intermediate-level semantic feature vector and the high-level semantic feature vector are combined in pairs to obtain multi-scale information fusion features;

[0018] The granular information extraction module consists of a 1×1 channel-wise convolutional layer, a 3×3 pointwise convolutional layer, a batch normalization layer, and a ReLU activation function.

[0019] In some embodiments of this application, the hierarchical feature mapping of the multi-scale fused feature vectors at each level includes:

[0020] The upper-level coarse-grained feature vector is mapped to its lower-level fine-grained feature vector using a feature mapping algorithm, denoted as . Where l represents the l-th layer of the hierarchical granularity;

[0021] The upper-level coarse-grained feature vectors are mapped onto the fine-grained feature vector space to obtain multi-granular feature information;

[0022] The mapped feature vector is fused with the mapped fine-grained feature vector.

[0023] In some embodiments of this application, the category-center-based triplet loss includes

[0024] Fine-grained feature extraction is performed on the input image to obtain the prediction result;

[0025] If the prediction is correct, TCL is not used; if the prediction is an incorrect category, the corresponding error category feature center vector is queried from the category feature center database. and the correct category feature center vector

[0026] The feature centers of the correct and incorrect categories are used as positive and negative samples, respectively. The fine-grained feature vector of the image extracted by the model is denoted as f. a Using triples Constituting the triplet loss:

[0027]

[0028] Where d(a, b) represents the calculation of the distance between a and b, and α represents the threshold;

[0029] Update the feature centers of the correct and incorrect categories in the category feature center database.

[0030] In some embodiments of this application, the loss function of the hierarchical multi-granularity image classification model further includes a reordering classification loss;

[0031] After obtaining the category prediction vector v, retain the maximum value of the first k category prediction probabilities, and set the other values ​​to 0;

[0032] Normalize the category prediction vector to obtain a new category prediction vector v. Top-k Among them, the intermediate semantic category prediction vector is denoted as High-level semantic category prediction vector, denoted as

[0033]

[0034] in, This means retaining the highest predicted probability for the first k categories, and setting all other values ​​to 0. k is a hyperparameter, and f... n (·) indicates normalization;

[0035] The Reclassification Loss (RCL) of intermediate semantic classification information can be defined as Equation (7), the RCL of high-level semantic classification information can be defined as Equation (8), and the total reclassification loss can be defined as Equation (9):

[0036]

[0037]

[0038]

[0039] Where i represents the i-th level of the hierarchical granularity, CE i (·) represents the cross-entropy loss of the i-th layer, label i This represents the actual label of the i-th layer.

[0040] In some embodiments of this application, the hierarchical multi-granularity image classification network is jointly trained using a reordering classification loss and a class center-based triplet loss, with the overall loss expressed as:

[0041] Loss = L RCL +L TCL .

[0042] Another aspect of this application provides a multi-granularity image classification device, implemented through a hierarchical multi-granularity image classification model, the classification device comprising:

[0043] The feature map acquisition module is used to acquire feature maps with multi-scale information.

[0044] The extraction mapping module is used to extract and fuse feature vectors at different levels of granularity to obtain multi-scale fused feature vectors at various levels.

[0045] The mapping module is used to perform hierarchical feature mapping on the multi-scale fused feature vectors of each level;

[0046] The fusion module is used to fuse the mapped feature vector with the mapped fine-grained feature vector.

[0047] The loss function of the hierarchical multi-granularity image classification model includes a triplet loss based on class center.

[0048] Another aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-granularity image classification method described in any embodiment of this application.

[0049] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program is executed by a processor to implement the multi-granularity image classification method described in any embodiment of this application.

[0050] One aspect of the technical solution provided in this application embodiment may include the following beneficial effects:

[0051] The multi-granularity image classification method provided in this application embodiment is implemented through a hierarchical multi-granularity image classification model to obtain feature maps with multi-scale information; extract and fuse feature vectors of different levels of granularity to obtain multi-scale fused feature vectors of each level; perform hierarchical feature mapping on the multi-scale fused feature vectors of each level; and perform a fusion operation between the mapped feature vectors and the mapped fine-grained feature vectors. The loss function of the hierarchical multi-granularity image classification model includes a triplet loss based on the class center, which increases the local detail information of the network and solves the problem of the lack of local detail information in the network. By fusing the multi-granular features of each class level, the problem of different regions corresponding to coarse and fine granularities between levels is solved. The use of triplet loss based on the class center enhances the feature representation of the image and improves the classification accuracy of images at each level of granularity.

[0052] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing embodiments of this application. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart of a multi-granularity image classification method according to an embodiment of this application is shown.

[0055] Figure 2 A schematic diagram of a hierarchical multi-granularity model structure according to an embodiment of this application is shown.

[0056] Figure 3 A schematic diagram of the structure of a hierarchical cross-fusion module according to an embodiment of this application is shown.

[0057] Figure 4 A schematic diagram of the reordering classification loss of one embodiment of this application is shown.

[0058] Figure 5 A bar chart showing the effect of the hyperparameter k of the reordering classification loss (RCL) on the CUB dataset is provided as an example.

[0059] Figure 6 A bar chart showing the effect of the hyperparameter k of the reordering classification loss (RCL) on the Air dataset is shown in one example.

[0060] Figure 7 A bar chart showing the effect of the hyperparameter k of the reordering classification loss (RCL) on the Car dataset is shown in one example.

[0061] Figure 8 A visualization of mid-to-high-level semantic feature maps of a baseline model and the method of this application embodiment at different levels is shown in an example.

[0062] The purpose, functional features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0064] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0065] The inventors of this application have discovered that different target regions should be considered when performing classification at different levels of granularity. This transformation between different regions corresponds to a mapping mechanism between features at different levels.

[0066] To uncover the detailed texture information of fine-grained images at different granularities and focus on the relationships between hierarchical features, this application proposes a hierarchical multi-granularity image classification method that integrates multi-scale information and feature mapping relationships. Addressing the problem of insufficient local detail information in feature extraction networks, this method employs a fusion of intermediate and high-level semantic features from the backbone network as multi-scale feature representations of images at different category granularities, thereby increasing the network's local detail information and resolving the issue of insufficient local detail information. To address the problem of different regions corresponding to coarse and fine granularities between levels, this application uses a feature mapping algorithm to represent the feature mapping relationships between different levels, fusing multi-granular features of each category level to resolve the issue of different regions corresponding to coarse and fine granularities between levels. This application also introduces a re-ordering classification loss (RCL) and a triplet center loss (TCL) based on category centers to enhance image feature representation and improve the classification accuracy of images at different granularities. Comparative experiments were conducted on three commonly used hierarchical multi-granularity image classification datasets. On multiple metrics, the method of this application achieved performance comparable to or even better than related techniques.

[0067] refer to Figure 1 As shown, one embodiment of this application provides a multi-granularity image classification method, implemented through a hierarchical multi-granularity image classification model. The classification method may include:

[0068] The process involves: acquiring feature maps with multi-scale information; extracting and fusing feature vectors at different levels of granularity to obtain multi-scale fused feature vectors at each level; performing hierarchical feature mapping on the multi-scale fused feature vectors at each level; and fusing the mapped feature vectors with the mapped fine-grained feature vectors. The loss function of the hierarchical multi-granularity image classification model includes a triplet loss based on the category center.

[0069] In some implementations, feature vectors at different levels of granularity are extracted and fused to obtain multi-scale fused feature vectors at each level, including: extracting granular information from high-level semantic feature maps to obtain high-level semantic feature vectors; extracting granular information from mid-level semantic feature maps to obtain mid-level semantic feature vectors; and combining mid-level semantic feature vectors and high-level semantic feature vectors in pairs to obtain multi-scale information fused features; wherein the granular information extraction module consists of a 1×1 channel-wise convolutional layer, a 3×3 pointwise convolutional layer, a batch normalization layer, and a ReLU activation function.

[0070] For example, the hierarchical multi-granularity image classification model comprises three parts: a feature encoding stage, a multi-scale feature extraction and fusion stage, and a hierarchical feature mapping stage.

[0071] Feature maps acquiring multi-scale information are achieved through the feature encoding stage. This application's embodiments use a ResNet-50 pre-trained on ImageNet-1K as the backbone network. For example... Figure 2 As shown, the feature map F obtained by taking the third block of ResNet-50 is... m ∈R C×2W×2H As intermediate semantic information, the feature map F obtained from the 4th block of ResNet-50 is used. h ∈R C×W×H As high-level semantic information, C, W, and H represent the number of channels, width, and height of the fourth feature map of the ResNet-50 backbone network, respectively. These two feature maps represent information at different scales and serve as inputs to the multi-granularity extraction and fusion stage.

[0072] The extraction and fusion of feature vectors at different levels of granularity to obtain multi-scale fused feature vectors at each level is achieved through a multi-scale feature extraction and fusion stage.

[0073] After obtaining feature maps with multi-scale information, a granularity information extraction module is used to extract feature vectors at different levels of granularity. Each granularity information extraction module is independent and does not share parameters; they extract feature vectors at different levels of granularity for feature maps of different scales. For example... Figure 2 As shown, the high-level semantic feature map F h The granularity feature vectors f at each level are obtained through the granularity information extraction module. hi Where i = 1, 2, 3; the larger the value of i, the finer the granularity. Intermediate semantic feature map F m The granularity feature vectors f at each level are obtained through the granularity information extraction module. mi Where i = 1, 2, 3. The granular information extraction module consists of a 1×1 channel-wise convolutional layer, a 3×3 pointwise convolutional layer, a batch normalization (BN) layer, and a ReLU activation function, obtaining granular feature vectors at each level:

[0074] f mi =Ψ 3x3 (Ψ 1×1 (F mi )), i = 1, 2, 3 (1)

[0075] f hi =Ψ 3×3 (Ψ 1×1 (F hi )), i = 1, 2, 3 (2)

[0076] Where i represents the i-th level of the hierarchical granularity, Ψ 1×1This indicates that the circuit first passes through a 1×1 channel-wise convolutional layer, then a batch normalization layer, and finally a ReLU activation function. Ψ 3×3 This means that the circuit first passes through a 3×3 channel-wise convolutional layer, then through a batch normalization layer, and finally through the ReLU activation function.

[0077] Then, based on different levels of granularity, the obtained intermediate-level semantic feature vectors and high-level semantic feature vectors are combined pairwise to obtain multi-scale information fusion features:

[0078] f i =Concat(f hi f mi ), i = 1, 2, 3 (3)

[0079] Here, Concat(a, b) represents the concatenation operation between vectors a and b. The resulting multi-scale fused feature vectors at each level are used as inputs to the hierarchical feature mapping stage.

[0080] The hierarchical feature mapping is performed on the multi-scale fused feature vectors at each level, and the fusion operation between the mapped feature vectors and the mapped fine-grained feature vectors is achieved through the hierarchical feature mapping stage.

[0081] To focus on the relationships between feature vectors at different levels after fusing multi-scale information, the feature vectors obtained from the multi-scale feature extraction and fusion stages are input into the hierarchical cross-fusion module. For example... Figure 3 As shown, in the same feature space, the hierarchical cross-fusion module maps the upper-layer coarse-grained feature vectors to the lower-layer fine-grained feature vectors using a feature mapping algorithm, denoted as... Here, l represents the l-th layer of hierarchical granularity. The upper-layer coarse-grained feature vectors are mapped onto the fine-grained feature vector space to obtain multi-granular feature information. Then, the mapped feature vectors are fused with the mapped fine-grained feature vectors. The hierarchical feature mapping and fusion operations can be defined as follows:

[0082]

[0083]

[0084] Where BN(·) represents the batch normalized layer and LN(·) represents the linear layer.

[0085] The loss function of a hierarchical multi-granularity image classification model can also include re-classification loss.

[0086] To enhance the robustness of feature extraction by the hierarchical multi-granularity model, this application proposes two loss functions: Reordering Classification Loss (RCL) and Triple Loss Based on Class Center (TCL).

[0087] During classification, the higher the predicted probability value of a specific category, the more relevant it is to the correct category and the easier it is to confuse; conversely, the lower the predicted probability value of a category, the less relevant it is to the correct category. The existence of these irrelevant predicted values ​​will negatively affect the calculation of the final classification loss. Therefore, it is necessary to suppress erroneous category information similar to the correct category, allowing the network to focus on distinguishing easily confused categories. Based on the above conclusions, this application proposes a Reordering Classification Loss (RCL). Figure 4 As shown, after obtaining the class prediction vector v, the maximum value of the first k class prediction probabilities is retained, and the other values ​​are set to 0. Then, the class prediction vector is normalized to obtain a new class prediction vector v. Top-k Among them, the intermediate semantic category prediction vector is denoted as High-level semantic category prediction vector, denoted as

[0088]

[0089] in, This means retaining the highest predicted probability for the first k categories, and setting all other values ​​to 0. k is a hyperparameter, and f... n (·) indicates normalization.

[0090] The Reclassification Loss (RCL) of intermediate semantic classification information can be defined as Equation (7), the RCL of high-level semantic classification information can be defined as Equation (8), and the total reclassification loss can be defined as Equation (9):

[0091]

[0092]

[0093]

[0094] Where i represents the i-th level of the hierarchical granularity, CE i (·) represents the cross-entropy loss of the i-th layer, label i This represents the actual label of the i-th layer.

[0095] In the FGIC task, the similarity between images of different fine-grained categories may be higher than the similarity between images within the same category. Therefore, embodiments of this application employ triplet loss (TCL) based on category centers. Figure 2 As shown, the feature vectors of the input image training set in each round are stored in the category feature center database after training, and the feature vectors of the previous round are updated in the next round. After each round of training, the category feature center database calculates and stores the feature center vector for each category.

[0096] The model performs fine-grained feature extraction on the input image to obtain the prediction result. When the prediction result is correct, TCL is not used; when the prediction result is an incorrect category, the model queries the category feature center database for the corresponding incorrect category feature center vector. and the correct category feature center vector The feature centers of the correct and incorrect categories are used as positive and negative samples, respectively. The fine-grained feature vector of the image extracted by the model is denoted as f. a The three feature vectors mentioned above form a triplet. This constitutes a triplet loss:

[0097]

[0098] Where d(a, b) represents the distance between a and b, and α represents the threshold used to update the feature centers of the correct and incorrect classes in the category feature center database. In this embodiment, α is set to 0.3. To stabilize network training, the feature storage and updating of the category feature center database begins in the middle of training.

[0099] The hierarchical multi-granularity image classification network, which integrates multi-scale information and feature mapping relationships, is jointly trained using the aforementioned re-ranking classification loss and class center-based triplet loss. The final overall loss can be expressed as:

[0100] Loss = L RCL +L TCL (11).

[0101] In a specific example, the method of this application embodiment was tested on three hierarchical multi-granularity datasets: CUB-200-2011, FGVC-Aircraft, and Stanford Cars. The dataset information is shown in Table 1.

[0102] CUB-200-2011 (CUB) is a bird dataset containing 11,877 images of 200 bird species. The dataset has been reorganized into a three-level label hierarchy with 13 orders (e.g., "Passeriformes" and "Charadriiformes"), 38 families (e.g., "Diomedeidae" and "Alcidae"), and 200 species (e.g., "Black footed Albatross" and "Parakeet Auklet").

[0103] FGVC-Aircraft (Air) is an aircraft dataset containing 10,000 images covering 100 aircraft models. It has a three-level label hierarchy, including 30 manufacturers (e.g., "Boeing" and "Douglas Aircraft"), 70 series (e.g., "Boeing 767", "Boeing 777"), and 100 models (e.g., "767-400", "767-500").

[0104] Stanford Cars (Car) contains images of 8,144 cars across 196 models. The dataset has been reorganized into a two-level label hierarchy with 9 car types (e.g., “Cab” and “SUV”) and 196 specific models (e.g., “Mazda Tribute SUV 2011” and “Infiniti QX56 SUV 2011”).

[0105] The embodiments in this application follow the standard training and testing splitting methods listed in the original dataset. No additional annotation information is used in any of the experiments.

[0106] Table 1 Experimental Dataset

[0107]

[0108] During the training phase, this embodiment uses SGD for network optimization, training for a total of 200 rounds. The initial learning rate of the backbone network ResNet-50 is set to 4×10⁻⁶. -4 The initial learning rate for other modules is set to 4×10. -3 After training for 80 rounds, in each round, the category feature center database updates the feature vector of each category from the previous round, and a triplet loss based on category centers is added to help the network learn better. This embodiment uses horizontal flipping and center cropping as data augmentation, with an image input size of 448×448.

[0109] During the testing phase, the input image size is set to 448×448. After the feature extraction phase, high-level semantic feature maps are extracted. Then, after the multi-granularity feature extraction and fusion phase, feature vectors containing high-level semantic information are obtained. Finally, after the hierarchical feature mapping phase, classification features at each level of granularity are obtained, and then classification prediction is performed.

[0110] All experiments in this application were coded using PyTorch and conducted on a server configured with an Intel(R) CPU 2.20GHz and an NVIDIA A40 GPU.

[0111] This application uses two evaluation metrics. The first evaluation metric follows the FGIC convention and uses fine-grained accuracy for evaluation. The second metric is to calculate the classification accuracy for all hierarchical granularities. Then, the hierarchical classification performance can be evaluated by calculating the weighted average precision (wAP) of all hierarchical structures, which can be defined by equation (12):

[0112]

[0113] Where class_num l P represents the number of categories in the l-th layer. l This represents the Top-1 accuracy of layer l. Because fine-grained layers have more classes, they have a greater weight in wAP performance evaluation than coarse-grained classification accuracy.

[0114] Table 2 Comparison of performance with existing hierarchical multi-granularity image classification (%)

[0115]

[0116] Table 3. Ablation experiments of different components of the hierarchical multi-granular model integrating multi-scale information and feature mapping relationships on CUB.

[0117]

[0118] To verify the effectiveness of the method in this application, it was compared with existing methods. As shown in Table 2, the Baseline includes a basic structure: a backbone network ResNet-50, a granular information extraction module, and a classification layer corresponding to each level. Baseline* indicates that the network structure only includes the backbone network ResNet-50 and a fine-grained classification layer. HMC-LMLP, HMCN, and C-HMCNN focus on the impact of coarse-grained level prediction scores on fine-grained level classification. FGN studies the impact of transfer between different granularity classification tasks. HRN focuses on the correlation between levels and the mutual exclusion of the same level category. The method in this application focuses on the impact of multi-scale information on hierarchical multi-granularity image classification and the feature mapping relationship between different granularity levels.

[0119] On the one hand, the fine-grained classification accuracy using the hierarchical multi-granularity classification method (Baseline) is higher than that without the hierarchical multi-granularity classification method (Baseline*). Specifically, on the CUB, Air, and Car datasets, the fine-grained accuracy of Baseline is 85.4%, 91.5%, and 93.5%, respectively, all of which are superior to Baseline*. Therefore, experiments demonstrate that hierarchical multi-granularity classification can help achieve better performance in fine-grained image classification tasks. Furthermore, although Baseline uses hierarchical multi-granularity classification, the levels are independent of each other and lack information interaction. By adding the multi-scale information fusion module proposed in the embodiments of this application, the fine-grained classification accuracy can be further improved.

[0120] On the other hand, experiments demonstrate that on the CUB dataset, the classification accuracy and wAP of the method in this embodiment are 98.9%, 95.9%, 88.8%, and 90.4% at each level, respectively, all of which outperform existing methods. On the Air and Car datasets, the classification accuracy of the method in this embodiment at the finest granularity is 94.1% and 95.1%, respectively, higher than existing methods. The P2 and wAP are equal for Car because P1 has 9 classes, which has a small proportion in the wAP calculation and thus a minimal impact. The overall classification accuracy wAP values ​​show that the method in this embodiment outperforms existing methods on all three datasets. Overall, the method in this embodiment achieves a high level of performance compared to numerous comparative models.

[0121] To verify the effectiveness of the various components of the hierarchical multi-granularity model that integrates multi-scale information and feature mapping relationships, ablation experiments were conducted, as shown in Table 3. Scheme 1 represents the fine-grained accuracy of the baseline and is used as the benchmark. Analysis of the results in the table shows that:

[0122] (1) Scheme 2 demonstrates that after fusing high-level semantic features with mid-level semantic features that incorporate detailed textures, the fine-grained accuracy is 87.4%. By acquiring more local detailed texture information, the fine-grained classification performance can be significantly improved.

[0123] (2) Scheme 3 uses a hierarchical cross-fusion module, which improves the fine-grained accuracy by 1.4% compared with the baseline. Through the cross-mapping relationship between the levels, the fine-grained level can obtain richer coarse-grained information, thus improving the classification performance of the fine-grained level.

[0124] (3) Scheme 4 demonstrates that adding reordering classification loss can improve fine-grained accuracy by 1.1% compared to the baseline. By filtering out noisy categories, classification accuracy can be effectively improved.

[0125] (4) Scheme 5 demonstrates that adding a class center-based triplet loss can improve fine-grained accuracy by 1.3% compared to the baseline. By specifically strengthening the contrastive learning of positive and negative samples, the network learns more discriminative feature representations, thus improving the fine-grained classification performance.

[0126] (5) Schemes 6, 7 and 8 demonstrate that the best fine-grained classification results are achieved when the combination of mid-to-high-level feature fusion module + hierarchical cross fusion module + reordering classification loss + class center-based triplet loss is used, with a fine-grained classification accuracy of 88.8%.

[0127] Because the evaluation criteria for this application embodiment are fine-grained classification accuracy and wAP, this application embodiment focuses on testing the impact of different values ​​of the hyperparameter k of the re-ranking classification loss (RCL) on model performance in fine-grained classification. Figure 5 As shown, on the CUB dataset, the model achieved the best performance when k=8, with a fine-grained classification accuracy of 88.8% and a wAP of 90.4%. Figure 6 As shown, on the Air dataset, the fine-grained classification accuracy for model numbers is highest at 94.2% when k=9, and the wAP is highest at 95.1% when k=8; Figure 7 As shown, on the Car dataset, when k=7, the fine-grained classification accuracy of the model is the highest at 95.1%, and the wAP is the highest at 95.1%.

[0128] A comparative experiment was conducted on the parameter count and computational cost of the hierarchical multi-granularity model integrating multi-scale information and feature mapping relationships in this application embodiment, and existing hierarchical multi-granularity models. As shown in Table 4, FGN and HRN are the two best-performing hierarchical multi-granularity image classification models currently available. Our* indicates that the granularity information extraction module in the Our model designed in this application embodiment is replaced by depthwise separable convolution, reducing the model parameter count and computational cost. Experiments show that the Our model has a higher parameter count than the FGN model but a lower parameter count than the HRN model, but its wAP performance is better than both existing methods. The Our* model achieves a good balance between model parameter count and model performance. In summary, when focusing on improving wAP performance, the Our model method can be selected; when focusing on both model parameter count and model performance, the Our* model method can be selected.

[0129] Table 4 compares the parameter count (M), computational cost (G), and wAP (%) of existing hierarchical multi-granularity image classification models.

[0130]

[0131] The above experiments demonstrate that hierarchical multi-granularity image classification is helpful for fine-granular image classification tasks. Therefore, in this section, the method of this application embodiment is compared with some fine-granular image classification methods in recent years. As shown in Table 5, for the Air and Car datasets, the method of this application embodiment outperforms existing fine-granular image classification methods, with fine-granular classification accuracies of 94.2% and 95.1%, respectively. For the CUB dataset, the method of this application embodiment also achieves results comparable to the DF-GMM method. Compared with fine-granular image classification methods in recent years, the method of this application embodiment utilizes the fusion of multi-scale information and the feature inter-mapping relationships between different hierarchical structures to improve performance and achieve competitive results.

[0132] Table 5 compares the performance of fine-grained image classification methods in recent years.

[0133]

[0134] For qualitative analysis, this example also compares the visualization results of the response regions of the baseline model and the mid-to-high-level semantic feature maps. For example... Figure 8 As shown, high-level semantic features focus more on the object itself; as the granularity increases, the area of ​​focus becomes smaller and more discriminative. Meanwhile, mid-level semantic features focus more on the texture features of the object. Combining mid- and high-level semantic information enhances the network's representational ability, thereby more effectively improving the performance of hierarchical multi-granularity image classification. Figure 8 As can be seen, on the CUB dataset, the method of this application focuses on the object itself in the "eye" category, without being disturbed by background noise, indicating that the features extracted by the network have stronger robustness.

[0135] The method proposed in this application achieves hierarchical multi-granularity image classification by fusing multi-scale information and feature mapping relationships. Extensive experiments on three commonly used fine-grained image classification datasets demonstrate that the proposed method exhibits significant effectiveness and state-of-the-art performance in hierarchical multi-granularity image classification tasks. The re-ranking classification loss and the class center-based triplet loss can better assist the network in learning discriminative feature information.

[0136] Another embodiment of this application provides a multi-granularity image classification device, implemented through a hierarchical multi-granularity image classification model, the classification device comprising:

[0137] The feature map acquisition module is used to acquire feature maps with multi-scale information.

[0138] The extraction mapping module is used to extract and fuse feature vectors at different levels of granularity to obtain multi-scale fused feature vectors at various levels.

[0139] The mapping module is used to perform hierarchical feature mapping on the multi-scale fused feature vectors of each level;

[0140] The fusion module is used to fuse the mapped feature vector with the mapped fine-grained feature vector.

[0141] The loss function of the hierarchical multi-granularity image classification model includes a triplet loss based on class center.

[0142] Another embodiment of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the above embodiments.

[0143] The electronic devices and methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0144] Another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the methods described in any of the above embodiments.

[0145] It should be noted that examples of computer-readable storage media may include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0146] The computer-readable storage medium provided in the above embodiments of this application and the method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0147] It should be noted that the term "module" is not intended to be limited to a specific physical form. Depending on the specific application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. Furthermore, different modules may share common components or even be implemented using the same components. Clear boundaries may or may not exist between different modules.

[0148] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used with the examples based on this. The required structure for constructing such devices is obvious from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0149] It should be understood that although the steps in the flowchart of the accompanying drawings are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in the embodiments of this application, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0150] The above embodiments merely illustrate the implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-granularity image classification method, characterized in that, This is achieved through a hierarchical multi-granularity image classification model, and the classification method includes: Obtain feature maps with multi-scale information; Feature vectors at different levels of granularity are extracted and fused to obtain multi-scale fused feature vectors at each level. Granular information is extracted from the high-level semantic feature map to obtain the high-level semantic feature vector. Granular information is extracted from the mid-level semantic feature map to obtain the mid-level semantic feature vector. The mid-level semantic feature vector and the high-level semantic feature vector are combined pairwise to obtain the multi-scale information fused feature. The granular information extraction module consists of a 1×1 channel-wise convolutional layer, a 3×3 pointwise convolutional layer, a batch normalization layer, and a ReLU activation function. Hierarchical feature mapping is performed on the multi-scale fused feature vectors at each level, mapping the upper-level coarse-grained feature vectors to the lower-level fine-grained feature vectors using a feature mapping algorithm, denoted as . Where l represents the l-th layer of the hierarchical granularity, the upper coarse-grained feature vector is mapped to the fine-grained feature vector space to obtain multi-granular feature information; The mapped feature vector is fused with the mapped fine-grained feature vector. The loss function of the hierarchical multi-granularity image classification model includes a triplet loss based on class centers. Fine-grained feature extraction is performed on the input image to obtain the prediction result. When the prediction result is correct, the triplet loss (TCL) is not used; when the prediction result is an incorrect class, the corresponding incorrect class feature center vector is queried from the class feature center database. and the correct category feature center vector The feature centers of the correct and incorrect categories are used as positive and negative samples, respectively. The fine-grained feature vector of the image extracted by the model is denoted as f. a Using triples Constituting the triplet loss: Where d(a,b) represents calculating the distance between a and b, α represents the threshold, and the feature centers of the correct and incorrect categories are updated in the category feature center database; The loss function of the hierarchical multi-granularity image classification model also includes a re-sorting loss. After obtaining the class prediction vector v, the maximum value of the first k class prediction probabilities is retained, and the other values ​​are set to 0. The class prediction vector is then normalized to obtain a new class prediction vector v. Top-k Among them, the intermediate semantic category prediction vector is denoted as High-level semantic category prediction vector, denoted as in, This means retaining the highest predicted probability for the first k categories, and setting all other values ​​to 0. k is a hyperparameter, and f... n (·) indicates normalization; the re-ranking classification loss RCL for intermediate semantic classification information can be defined as Equation (1), the RCL for high-level semantic classification information can be defined as Equation (2), and the total re-ranking classification loss can be defined as Equation (3): Where i represents the i-th level of the hierarchical granularity, CE i (·) represents the cross-entropy loss of the i-th layer, label i This represents the actual label of the i-th layer.

2. The method according to claim 1, characterized in that, The hierarchical multi-granularity image classification model uses a ResNet-50 pre-trained on ImageNet-1K as its backbone network. The feature map for obtaining multi-scale information includes: The feature map F obtained from the third block of ResNet-50 m ∈R C×2W×2H As intermediate semantic information; the feature map F obtained from the 4th block of ResNet-50 is taken. h ∈R C×W×H As high-level semantic information, C, W, and H represent the number of channels, width, and height of the fourth feature map of the backbone network ResNet-50, respectively.

3. The method according to claim 1, characterized in that, The hierarchical multi-granularity image classification network is jointly trained using reordering classification loss and class center-based triplet loss, with the overall loss expressed as: Loss = L RCL +L TCL .

4. A multi-granularity image classification device, characterized in that, The method described in claim 1 is implemented through a hierarchical multi-granularity image classification model, wherein the classification device comprises: The feature map acquisition module is used to acquire feature maps with multi-scale information. The extraction mapping module is used to extract and fuse feature vectors at different levels of granularity to obtain multi-scale fused feature vectors at various levels. The mapping module is used to perform hierarchical feature mapping on the multi-scale fused feature vectors of each level; The fusion module is used to fuse the mapped feature vector with the mapped fine-grained feature vector. The loss function of the hierarchical multi-granularity image classification model includes a triplet loss based on class center.

5. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor to implement the method as described in any one of claims 1-3.

Images