A Transformer-based fine-grained image classification method and system

Abstract

The present invention discloses a Transformer-based fine-grained image classification method and system. The system includes a detail enhancement submodule that can enhance the model's ability to capture key details, a local feature refinement submodule that can focus on key areas of the input image by designing a channel attention mechanism and a spatial attention mechanism; and an adaptive feature fusion module that integrates multi-scale features and is more discriminative when processing highly similar images, so that the solution provided by the invention realizes the optimized integration of feature information, improves the expressiveness of features, enhances the model's generalization ability for complex images, and can effectively improve the accuracy and robustness of fine-grained image classification.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a Transformer-based fine-grained image classification method and system. Background Art

[0002] Fine-grained visual classification (FGVC) is a challenging task in computer vision, which involves classifying images into very specific and detailed categories, such as different species of birds, dogs, vehicle models, and medical images. Figure 1 As shown in the figure, the appearance of these four sparrows is almost the same, but the appearance of the same sparrow is also very different from different angles. Fine-grained image classification has a wide range of applications in real life. For example, in the field of security monitoring, fine-grained image classification can be used for tasks such as face recognition and behavior analysis; in the field of intelligent transportation, fine-grained image classification can be used for tasks such as vehicle brand and model recognition. Therefore, the research on fine-grained image classification has important theoretical and practical significance.

[0003] The goal of FGVC in computer vision is to retrieve and recognize images belonging to multiple subcategories of a supercategory (aka meta-category or base category), e.g., different species of animals / plants, different models of cars, different kinds of retail products, etc. Therefore, the key challenge lies in understanding fine-grained visual differences to fully distinguish objects that are highly similar in overall appearance but different in fine-grained features. Since its inception nearly 20 years ago, tremendous progress has been made. Given the effectiveness of convolutional neural networks (CNNs) in image recognition, existing image classification methods are generally based on convolutional neural network models, with optimization processing in image acquisition, data preprocessing, and dataset construction to improve the accuracy of image classification. However, traditional CNN architectures have limitations in processing large-scale image datasets, are insufficient in capturing long-range dependencies, and lose a lot of spatial representations during the continuous convolution and pooling process, making it impossible to obtain strong discriminative features. Summary of the invention

[0004] In order to solve the technical problems of low recognition accuracy and low efficiency in the existing methods for recognizing extremely fine features of images, an embodiment of the present invention provides a Transformer-based fine-grained image classification method and system.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] The embodiment of the present invention provides a Transformer-based fine-grained image classification system, the system includes a fine-grained image classification model and a fully connected layer, the fine-grained image classification model includes a backbone network, a feature pyramid network and a fine-grained feature extraction module; the fine-grained feature extraction module includes a detail enhancement submodule, a local feature refinement submodule and an adaptive feature fusion module; the backbone network is used to process multi-scale information in an input image through a built-in moving window mechanism, and output a multi-scale feature map of the input image; the feature pyramid network is a deep learning model, which is used to perform multi-scale feature extraction and fusion on the multi-scale feature map output by the backbone network by constructing a feature pyramid, and output a spatial domain feature map; the detail enhancement submodule is used to convert the spatial domain feature map output by the feature pyramid network to the frequency domain by using a discrete cosine transform to obtain a frequency domain feature map; high-frequency domain features for distinguishing the fine-grained features of the input image are screened out from the frequency domain feature map by an adaptive threshold; and the high-frequency domain features are converted back to the spatial domain by using a discrete cosine transform , and superimpose the converted spatial domain features with the spatial domain feature map output by the feature pyramid network, and output the feature map after detail enhancement; the local feature refinement submodule is used to construct a channel attention mechanism and a spatial attention mechanism, and use the channel attention mechanism to assign different weights to each channel, and use the spatial attention mechanism to assign different weights to each spatial position, and output the feature map after local feature refinement based on the channel-level attention map calculated by the channel attention mechanism, the spatial-level attention map calculated by the spatial attention mechanism, and the feature map after detail enhancement output by the detail enhancement submodule; the adaptive feature fusion module is used to obtain information from the multi-scale feature map output by the backbone network, the feature map after detail enhancement output by the detail enhancement submodule, and the feature map after local feature refinement output by the local feature refinement submodule, and dynamically adjust the fusion weight of each feature map through an adaptive weight learning mechanism to output the fusion feature; the fully connected layer is used to output the prediction result according to the fusion feature output by the adaptive feature fusion module.

[0007] The embodiment of the present invention also provides a Transformer-based fine-grained image classification method, which is applied to the Transformer-based fine-grained image classification system described above, and the method includes: using the backbone network to process multi-scale information in the input image through a built-in moving window mechanism, and outputting a multi-scale feature map of the input image; using the feature pyramid network to perform multi-scale feature extraction and fusion on the multi-scale feature map output by the backbone network by constructing a feature pyramid, and outputting a spatial domain feature map; using the detail enhancement submodule to convert the spatial domain feature map output by the feature pyramid network to the frequency domain by using a discrete cosine transform to obtain a frequency domain feature map; filtering out high-frequency domain features for distinguishing the fine-grained features of the input image from the frequency domain feature map by using an adaptive threshold; using a discrete cosine transform to convert the high-frequency domain features back to the spatial domain, and comparing the converted spatial domain features with the spatial domain feature map output by the feature pyramid network. The local feature refinement submodule is used to construct a channel attention mechanism and a spatial attention mechanism, and the channel attention mechanism is used to assign different weights to each channel, and the spatial attention mechanism is used to assign different weights to each spatial position. Based on the channel-level attention map calculated by the channel attention mechanism, the spatial-level attention map calculated by the spatial attention mechanism, and the detail-enhanced feature map output by the detail enhancement submodule, the feature map after local feature refinement is output; the adaptive feature fusion module is used to obtain information from the multi-scale feature map output by the backbone network, the detail-enhanced feature map output by the detail enhancement submodule, and the feature map after local feature refinement output by the local feature refinement submodule, and the fusion weight of each feature map is dynamically adjusted through an adaptive weight learning mechanism to output fusion features; the fully connected layer is used to output a prediction result according to the fusion features output by the adaptive feature fusion module.

[0008] An embodiment of the present invention further provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0009] This embodiment has the following beneficial effects:

[0010] The fine-grained feature extraction module proposed in this embodiment enhances the model's ability to capture key detail features in the image by combining frequency domain analysis and multi-scale convolutional networks; the adaptive feature fusion module achieves optimal integration of feature information by dynamically adjusting the fusion weights of feature maps of different scales, thereby improving the expressiveness of features and enhancing the model's generalization ability for complex images. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1This is a schematic diagram of an example of a fine-grained category image of a sparrow according to an embodiment of the present invention;

[0012] Figure 2 Schematic diagram of the structure of a Transformer-based fine-grained image classification system according to an embodiment of the present invention;

[0013] Figure 3 Schematic diagram of the FIC-Transformer model structure and process according to an embodiment of the present invention;

[0014] Figure 4 This is a schematic diagram of a DESM module according to an embodiment of the present invention;

[0015] Figure 5 Schematic diagram of the process of a Transformer-based fine-grained image classification method according to an embodiment of the present invention;

[0016] Figure 6 The figure is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Before introducing this embodiment, the following contents are introduced:

[0018] There are two main approaches to the research on fine-grained image classification, namely feature extraction and target detection methods.

[0019] 1. Feature extraction method

[0020] In the field of fine-grained image classification, there are two approaches to extract discriminative features from subtle regions, which can be roughly divided into partial object-based methods and attention-based methods.

[0021] The partial object-based method uses the model to generate candidate regions, find the local regions of the objects to be identified, and then extract discriminative features from them. MACNN trains positioning and classification accuracy by clustering feature maps into target parts. This unsupervised classification enhances feature learning by dividing patterns into object parts. S3N enhances features by finding local extreme values ​​of each category response on the feature map. In addition, WS-DAN enhances data by removing local extreme values ​​to discover other discriminative features. The positioning modules of these methods often require a large number of parameters to obtain accurate local positioning results, and the training complexity is high.

[0022] Attention-based methods use attention mechanisms to enhance feature learning and locate object details. MAMC generates multiple sets of features enhanced by attention mechanisms, and Cross-X uses attention maps from multi-excitation models to learn features from different categories. API-Net and PCA-Net use two images as input and calculate the attention between feature maps to enhance discriminative representation. CAP calculates the self-attention map of the output feature to represent the relationship between feature pixels, and SR-GNN uses a graph convolutional neural network to describe the relationship between parts. CAL adds counterfactual interventions to the attention map to predict categories.

[0023] With the development of Transformer in computer vision, many improved VisionTransformer architectures have been proposed, such as FFVT, SIM-Trans, TransFG, and AFTrans. These methods use the self-attention map in the Transformer layer to enhance feature learning and locate object details. Transformer was originally proposed by Vaswani et al. in the field of natural language processing. Its core idea is to model long-distance dependencies in sequence data through the self-attention mechanism. Subsequently, Transformer was successfully applied to the field of computer vision, and a series of Transformer-based visual models were born, such as Vision Transformer (ViT) and Swin Transformer. ViT is a Transformer-based image classification model that divides the input image into patches of fixed size and embeds these patches into sequences for processing by a standard Transformer encoder. Swin Transformer proposes a windowed self-attention mechanism that can capture local and global contextual information while maintaining computational efficiency. Swin Transformer gradually expands the receptive field through a hierarchical structure, enabling it to achieve excellent results in a variety of visual tasks.

[0024] Although Transformer has achieved great success in the field of image recognition, there are still some challenges to be solved. For example, ViT requires a large amount of data to converge during training and is prone to overfitting on small-scale datasets. In addition, although Swin Transformer solves the problem of computational efficiency through a windowing mechanism, it still has limitations when processing fine-grained features in images. For example, in fine-grained image classification tasks, subtle differences in text and patterns often determine the judgment of fine-grained features, and capturing these features is still a major challenge for the existing Transformer architecture.

[0025] 2. Target Detection Methods

[0026] The object detection method aims to find the location and classification of the object. The overall architecture and idea are similar to the fine-grained visual classification task. The difference is that the purpose of fine-grained visual classification is not to find the area of ​​the object, but to find the area with distinguishable features and whether these features can be used more effectively for recognition.

[0027] Supervised object detection methods often have better results. Faster-RCNN predicts whether each pixel position on the feature map is an object through the Region Proposal Network (RPN), and then predicts the category of the target area. YOLO and RetinaNet complete the prediction of position and category through the entire network. The above methods all learn the area of ​​the object from manual annotations to complete the recognition of the object area. The weakly supervised object detection (WSOD) method is used as an alternative to overcome these limitations. B.Zhou et al. observed that by learning the object class label, the representation of the object in space can be learned, that is, the object position is virtually annotated through the information on the feature map. For example, WCCN first completes the division of candidate regions through the Class Activation Map (CAM). The principle of CAM is to generate a class-specific heat map by performing a map prediction score on the previous feature map, and then use the second-stage model to screen better candidate regions. Methods such as ACoL and SPG are also based on CAM to complete positioning. WSOD2 scores virtual candidate boxes through top-down and bottom-up methods, and the highest score is used as the target output of the next layer. MIST refines the ROI through self-training, while WSCL improves the features of the ROI through data augmentation and contrastive learning. These methods use the output of the previous stage as the target and gradually discover the entire target object through the refinement process.

[0028] The above methods show that category labels can provide rich target localization features and locate more category classification areas. However, the goal of fine-grained image classification tasks is no longer to detect complete objects, but to find the most critical areas and use these areas for better discrimination.

[0029] Based on this, this embodiment proposes a deep learning model based on the Transformer architecture. This model can significantly improve the accuracy and robustness of identification through fine-grained feature extraction and adaptive feature fusion strategies. The fine-grained feature extraction module can enhance the model's ability to capture key details, while the adaptive feature fusion module optimizes the integration of multi-scale features. In addition, this embodiment improves the loss function for fine-grained classification tasks, further improving the model's learning effect on difficult samples and key area features.

[0030] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0031] The embodiment of the present invention provides a fine-grained image classification system based on Transformer, such as Figure 2 As shown, the system includes a fine-grained image classification model 101 and a fully connected layer 102, the fine-grained image classification model 101 includes a backbone network 201, a feature pyramid network 202 and a fine-grained feature extraction module 203; the fine-grained feature extraction module 203 includes a detail enhancement submodule 301, a local feature refinement submodule 302 and an adaptive feature fusion module 303;

[0032] The backbone network 201 is used to process multi-scale information in the input image through a built-in moving window mechanism and output a multi-scale feature map of the input image;

[0033] The feature pyramid network 202 is a deep learning model, which is used to extract and fuse multi-scale features of the multi-scale feature map output by the backbone network 201 by constructing a feature pyramid, and output a spatial domain feature map;

[0034] The detail enhancement submodule 301 is used to convert the spatial domain feature map output by the feature pyramid network 202 to the frequency domain by using discrete cosine transform to obtain a frequency domain feature map; filter out high-frequency domain features for distinguishing fine-grained features of the input image from the frequency domain feature map by using an adaptive threshold; convert the high-frequency domain features back to the spatial domain by using discrete cosine transform, and superimpose the converted spatial domain features with the spatial domain feature map output by the feature pyramid network 202 to output a feature map after detail enhancement;

[0035] The local feature refinement submodule 302 is used to construct a channel attention mechanism and a spatial attention mechanism, use the channel attention mechanism to assign different weights to each channel, use the spatial attention mechanism to assign different weights to each spatial position, and output a feature map after local feature refinement based on the channel-level attention map calculated by the channel attention mechanism, the spatial-level attention map calculated by the spatial attention mechanism, and the feature map after detail enhancement output by the detail enhancement submodule 301;

[0036] The adaptive feature fusion module 303 is used to obtain information from the multi-scale feature map output by the backbone network 201, the feature map after detail enhancement output by the detail enhancement submodule 301, and the feature map after local feature refinement output by the local feature refinement submodule 302, dynamically adjust the fusion weight of each feature map through an adaptive weight learning mechanism, and output a fused feature;

[0037] The fully connected layer 102 is used to output a prediction result according to the fusion features output by the adaptive feature fusion module.

[0038] In view of the difficulty of fine-grained image classification, this embodiment focuses on the fine-grained feature extraction of key areas of the image and proposes a fine-grained image classification model 101 (Fine-Grained Image Classification based on Transformer, FIC-Transformer) based on the Transformer architecture, which integrates multi-scale features to achieve highly similar image recognition.

[0039] Specifically, the detail enhancement submodule 301 specifically uses the following calculation formula to output the feature map after detail enhancement:

[0040]

[0041] in, is the feature map after detail enhancement, It is the spatial domain feature map output by the feature pyramid network. is the adaptive threshold function, is a parameter to adjust the enhancement amplitude. is the frequency domain feature map obtained by discrete cosine transform, is the inverse discrete cosine transform operation.

[0042] The local feature refinement submodule 302 specifically uses the following calculation formula to output the feature map after local feature refinement:

[0043]

[0044] in, is the feature map after local feature refinement, is the feature map after detail enhancement, is a channel-level attention map calculated by the channel attention mechanism, is the spatial level attention map calculated by the spatial attention mechanism, represents the dot product operation, Represents element-wise multiplication.

[0045] The adaptive feature fusion module 303 specifically uses the following calculation formula to output the fusion feature:

[0046]

[0047] in, To fusion features, Represents the activation function, specifically the ReLU function, Indicates a connection operation. is the weight obtained by adaptive learning, is the multi-scale feature map output by the backbone network, is the feature map after detail enhancement, It is the feature map after local feature refinement.

[0048] The detail enhancement submodule 301 in this embodiment can enhance the model's ability to capture key details; the local feature refinement submodule 302 can focus on the key areas of the input image by designing a channel attention mechanism and a spatial attention mechanism; the adaptive feature fusion module 303 integrates multi-scale features and is more discriminative when processing highly similar images.

[0049] In addition, this embodiment also designs a hybrid loss function based on weighted cross entropy loss and region-sensitive loss, wherein the weighted cross entropy loss is used to assign high weights to false sample categories and low weights to true sample categories; the region-sensitive loss is used to assign high weights to key areas and low weights to non-key areas. The model can focus on both global and local fine-grained features during the learning process.

[0050] Specifically, the calculation expression of the hybrid loss function is:

[0051]

[0052] in, is the mixed loss function, and is the weight coefficient, is the weighted cross entropy loss, It is the area sensitive loss.

[0053] The calculation expression of the weighted cross entropy loss is:

[0054]

[0055] in, is the weighted cross entropy loss, For category The weight of is the number of samples, For sample Belongs to category The real label, For the model category The predicted probability of .

[0056] The calculation expression of the regional sensitivity loss is:

[0057]

[0058] in, is the regional sensitivity loss, It is a key area The weight of is an exponential function, which means that the sample Is it a region? , is the number of samples, is the basic loss function, is the number of critical regions, is the model’s predicted label for the sample, is the true label of the sample.

[0059] The hybrid loss function based on weighted cross entropy loss and region-sensitive loss proposed in this embodiment further improves the learning effect of the model on difficult samples and key area features by adjusting the loss calculation method according to the characteristics of fine-grained classification tasks.

[0060] In summary, this embodiment has the following beneficial effects:

[0061] 1. The fine-grained feature extraction module proposed in this embodiment enhances the model's ability to capture key detail features in the image by combining frequency domain analysis and multi-scale convolutional networks; the adaptive feature fusion module achieves optimal integration of feature information by dynamically adjusting the fusion weights of feature maps of different scales, thereby improving the expressiveness of features and enhancing the model's generalization ability for complex images.

[0062] 2. This embodiment proposes a hybrid loss function based on weighted cross entropy loss and region-sensitive loss. According to the characteristics of fine-grained classification tasks, the loss calculation method is adjusted to further improve the model's learning effect on difficult samples and key area features.

[0063] Below, the present application scheme will be described in detail based on a specific embodiment.

[0064] In view of the difficulty of fine-grained image classification, this embodiment focuses on the fine-grained feature extraction of key areas of the image and proposes a fine-grained image classification model based on Transformer architecture (Fine-Grained Image Classification based on Transformer, FIC-Transformer), which integrates multi-scale features to achieve highly similar image recognition.

[0065] 1. Model structure

[0066] The fine-grained image classification model FIC-Transformer designed in this embodiment is as follows Figure 3 As shown. The model consists of a backbone network (Swin Transformer), a feature pyramid network (FPN) and a customized fine-grained feature extraction module (FGFM), which aims to accurately distinguish the subtle differences in fine-grained features in key areas of the image. The backbone network uses Swin Transformer, which effectively processes multi-scale information in the image through its unique moving window mechanism, and can obtain multi-scale features of the input image. However, the multi-scale features obtained by Swin Transformer alone are still insufficient when processing subtle features in the image. To this end, this embodiment uses the feature pyramid network (FPN) to further enhance the multi-scale representation capability of the model. The feature pyramid network FPN is a deep learning model for computer vision tasks such as target detection and semantic segmentation. It realizes multi-scale feature extraction and fusion by constructing a feature pyramid, thereby improving the detection capability of targets of different sizes. The core of the FPN network is that it combines high-level semantic information with low-level detail information, and enhances the expression capability of features through top-down paths, bottom-up paths and lateral connections. FPN improves the model's ability to perceive features of various scales by fusing feature maps of different scales layer by layer, especially in the processing of high-resolution images. This multi-level feature fusion method enables the network to perform well in processing images with complex backgrounds and diverse textures.

[0067] The fine-grained feature extraction module FGFM consists of three sub-modules. Figure 3As shown in the figure, the training samples pass through SwinTransformer and FPN to obtain multi-scale image feature information, and then are processed by the FGFM module to remove useless feature noise, further obtain fine-grained feature information that is conducive to classification, and finally obtain the prediction result through the fully connected layer fc.

[0068] 2. Swin Transformer

[0069] The backbone network Swin Transformer is an innovative visual transformer architecture that effectively solves the inductive bias and computational complexity problems encountered by early visual transformers (ViT) when processing image recognition tasks by introducing a moving window mechanism. Compared with traditional single-window transformers, Swin Transformer can show stronger robustness under different image perturbations.

[0070] The basic idea of ​​Swin Transformer is to limit the traditional global self-attention calculation to non-overlapping local windows. This hierarchical local attention mechanism not only reduces the computational burden, but also enables the model to capture local features in a similar way to convolutional neural networks. Specifically, given an image segmented as non-overlapping windows, each containing Block area (patch), each patch is mapped to a feature vector, and the feature matrix within the window is defined as ,in is the feature dimension. In each window, the self-attention calculation formula can be simplified as:

[0071]

[0072] in, For the new feature representation, , , They are query matrix, key matrix and value matrix respectively. , , is a learnable weight matrix, is a scaling factor used to stabilize the attention learning process. This process is performed independently in each window, greatly reducing the computational complexity.

[0073] To further enhance the global vision of the model, Swin Transformer achieves cross-window information exchange by changing the position of the window in consecutive layers (i.e., the "shifted window" strategy). At the end of each stage, the window is shifted by half the window size so that the window of the next stage covers the new image area, thereby achieving the gradual propagation of feature information. This process can be expressed as:

[0074]

[0075] in, The operation realizes the translation of the window on the image, ensuring that the model can gradually obtain a wider range of contextual information. In addition, in order to maintain the multi-scale representation capability of the model, Swin Transformer performs a downsampling operation of feature resolution between specific layers, and reduces the feature map by merging windows. This process is accompanied by a linear projection operation to adapt to the change of feature dimension. The specific downsampling operation can be expressed as:

[0076]

[0077] in, It can be average pooling or maximum pooling to reduce the resolution of feature maps, while is the linear transformation matrix of the downsampling layer.

[0078] 3. Fine-grained feature extraction module

[0079] In order to enrich local information, deeply mine and strengthen key detail features in the image, this embodiment proposes a fine-grained feature extraction module. Figure 3 As can be seen, FGFM consists of three core submodules: Detail Enhancement Submodule (DESM), Local Feature Refinement Submodule (LFRSM) and Adaptive Feature Recombination Submodule (AFReSM). The following will explain in detail the working principle of each submodule and its role in the feature processing flow.

[0080] (1) Detail Enhancement Sub-Module (DESM)

[0081] In view of the shortcomings of existing methods in dealing with background interference and extracting subtle features, this embodiment designs a detail enhancement module DESM. This module is based on frequency domain analysis and adaptive filtering technology, such as Figure 4As shown in the figure, the feature map is first converted to the frequency domain using discrete cosine transform (DCT) to separate high-frequency details, which are often ignored but are crucial to distinguishing fine-grained features. The model selectively enhances the high-frequency components directly related to fine-grained feature judgment through adaptive threshold screening, and then converts it back to the spatial domain through inverse DCT and superimposes it with the original feature map to complete detail enhancement. This process not only improves the sensitivity to subtle differences, but also enhances the robustness of the model. The formula is expressed as:

[0082]

[0083] in, is the original spatial domain feature, is the adaptive threshold function, It is a parameter that adjusts the enhancement amplitude to ensure that the enhancement effect is not excessive or insufficient.

[0084] (2) Local Feature Refinement Module (LFRSM)

[0085] The Local Feature Refinement Submodule (LFRSM) aims to further explore the local structural features in the image. These subtle local features have an important impact on the effectiveness of classification. LFRSM further processes the feature map output from DESM through a multi-scale convolutional network. This embodiment designs the Channel Attention and Spatial Attention mechanisms to highlight the most discriminative local features in the feature map. The channel attention mechanism enhances the model's sensitivity to different feature channels by assigning different weights to each channel; while the spatial attention mechanism highlights the key areas in the image by assigning weights to each spatial position. The combination of this dual attention mechanism not only improves the model's ability to perceive details, but also enhances its ability to understand and express complex features, so that the model can classify more accurately when faced with various images.

[0086] The formula is as follows:

[0087]

[0088] in, It is the feature map obtained after DESM processing, which already contains the initial enhanced features. It is a channel-level attention map calculated by the channel attention mechanism. It is the spatial level attention map calculated by the spatial attention mechanism.

[0089] (3) Adaptive Feature Fusion Module (AFReSM)

[0090] In the final stage of the model, this embodiment designs an adaptive feature recombination submodule (AFReSM) to ensure that features of different scales can be effectively fused. AFReSM collects information from multi-scale feature maps extracted by Swin Transformer, DESM, and LFRSM, and dynamically adjusts the fusion weight of each feature map through an adaptive weight learning mechanism to ensure that the model can fully utilize feature information at all levels. Feature cascading and nonlinear mapping networks further optimize the expressive power of features, so that the final feature map not only has strong discrimination, but also retains rich detail information. This feature recombination strategy improves the classification performance of the model, especially when processing highly similar images.

[0091] The formula is:

[0092]

[0093] in, Represents the activation function, and selects the ReLU function as the activation function. The operation is to connect feature maps from different sources.

[0094] 4. Loss Function

[0095] In the task of fine-grained image classification, the design of loss function is crucial to improving model performance. In order to solve the problem of learning detailed features in fine-grained image classification, this embodiment designs weighted cross-entropy loss and region-sensitive loss based on the traditional cross-entropy loss function, and combines the two to form a hybrid loss function, which optimizes the learning process of the model.

[0096] (1) Weighted Cross-Entropy Loss

[0097] In classification tasks, due to the imbalance problem between fine-grained features, the standard cross entropy loss may not be able to effectively guide model learning. Therefore, this embodiment proposes a weighted cross entropy loss, which adjusts the loss function by assigning different weights to different categories, thereby reducing the impact of true samples on false samples during training. Specifically, for cases of class imbalance, false samples are often relatively few, and the correct identification of these samples is the key to improving model performance. Therefore, this embodiment assigns higher weights to false sample categories to ensure that the model pays more attention to these samples during the learning process, thereby improving the recognition accuracy of false samples. The formula is expressed as:

[0098]

[0099] in, For category The weight of is the number of samples, For sample Belongs to category The real label, For the model category The predicted probability of .

[0100] (2) Region-Sensitive Loss

[0101] In fine-grained images, some key local areas are crucial for classification tasks, but the features of these areas are often easily ignored by the model. In order to increase the model's attention to these key areas, this embodiment designs a region-sensitive loss function. In the loss calculation, higher weights are given to key local areas through regional importance factors, so that the model pays more attention to feature learning in these areas during training. Specifically, the region-sensitive loss function assigns different weights to multiple key areas in the image, thereby guiding the model to perform deeper feature extraction and learning in these areas. The formula is expressed as:

[0102]

[0103] in, It is a region The weight of is an exponential function, which means that the sample Is it a region? , is the basic loss function, is the number of key regions. Through this design, the region-sensitive loss enables the model to more accurately learn key region features related to fine-grained feature identification, significantly improving the model's discriminative ability.

[0104] (3) Hybrid Loss

[0105] Combining the above two loss functions, this embodiment proposes a hybrid loss function (Hybrid Loss), which further optimizes the learning effect of the model by combining weighted cross entropy loss and region-sensitive loss. The design of the hybrid loss function takes into account the learning needs of category imbalance and key areas, ensuring that the model can focus on both global and local fine-grained features during the learning process. The specific formula is:

[0106]

[0107] in, and is the weight coefficient that balances the two losses and is determined by cross-validation. Through the optimization of the hybrid loss function, the model can more accurately learn the features of different categories and key areas in the fine-grained image classification task, thereby effectively improving the classification performance of fine-grained images.

[0108] This embodiment proposes a fine-grained image classification model based on the Transformer architecture, which improves the accuracy and robustness of recognition through fine-grained feature extraction and adaptive feature fusion strategies. Among them, the DESM and LFRSM sub-modules combine frequency domain analysis and multi-scale convolutional networks to enhance the model's ability to capture key details of the image. The AFReSM module realizes the integration and optimization of feature information by dynamically adjusting the fusion weights of feature maps of different scales, improves the expressiveness of features, and enhances the model's generalization ability for complex images. In addition, this embodiment also designs weighted cross entropy loss and region-sensitive loss functions to further improve the model's learning effect on difficult samples and key area features.

[0109] The experimental results show that the method of this embodiment has better performance than other classic methods, with an accuracy rate of more than 98%, which can effectively improve the efficiency and accuracy of fine-grained image classification. That is, the model of this embodiment has advantages in accuracy, robustness, and training efficiency.

[0110] In addition, the embodiment of the present invention also provides a Transformer-based fine-grained image classification method, which is applied to the Transformer-based fine-grained image classification system described above. Figure 5 As shown, the method includes:

[0111] Step 501: Utilize the backbone network to process multi-scale information in the input image through a built-in moving window mechanism, and output a multi-scale feature map of the input image;

[0112] Step 502: using the feature pyramid network to construct a feature pyramid to perform multi-scale feature extraction and fusion on the multi-scale feature map output by the backbone network, and output a spatial domain feature map;

[0113] Step 503: The spatial domain feature map output by the feature pyramid network is converted to the frequency domain by using discrete cosine transform through the detail enhancement submodule to obtain a frequency domain feature map; high-frequency domain features for distinguishing fine-grained features of the input image are screened out from the frequency domain feature map by using an adaptive threshold; the high-frequency domain features are converted back to the spatial domain by using discrete cosine transform, and the converted spatial domain features are superimposed on the spatial domain feature map output by the feature pyramid network to output a feature map after detail enhancement;

[0114] Step 504: construct a channel attention mechanism and a spatial attention mechanism using the local feature refinement submodule, use the channel attention mechanism to assign different weights to each channel, use the spatial attention mechanism to assign different weights to each spatial position, and output a feature map after local feature refinement based on the channel-level attention map calculated by the channel attention mechanism, the spatial-level attention map calculated by the spatial attention mechanism, and the feature map after detail enhancement output by the detail enhancement submodule;

[0115] Step 505: using the adaptive feature fusion module to obtain information from the multi-scale feature map output by the backbone network, the feature map after detail enhancement output by the detail enhancement submodule, and the feature map after local feature refinement output by the local feature refinement submodule, dynamically adjust the fusion weight of each feature map through an adaptive weight learning mechanism, and output the fused feature;

[0116] Step 506: Utilize the fully connected layer to output a prediction result according to the fusion features output by the adaptive feature fusion module.

[0117] The above method provided in this embodiment belongs to the same concept as the above system embodiment. The specific implementation process is detailed in the system embodiment and will not be repeated here.

[0118] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the above method.

[0119] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). Specifically, in one embodiment, the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0120] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0126] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0128] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0129] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0130] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0131] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A Transformer-based fine-grained image classification system, characterized in that: The system includes a fine-grained image classification model and a fully connected layer, wherein the fine-grained image classification model includes a backbone network, a feature pyramid network and a fine-grained feature extraction module; the fine-grained feature extraction module includes a detail enhancement submodule, a local feature refinement submodule and an adaptive feature fusion module; The backbone network is used to process multi-scale information in the input image through a built-in moving window mechanism and output a multi-scale feature map of the input image; The feature pyramid network is a deep learning model, which is used to extract and fuse multi-scale features of the multi-scale feature map output by the backbone network by constructing a feature pyramid, and output a spatial domain feature map; The detail enhancement submodule is used to convert the spatial domain feature map output by the feature pyramid network into the frequency domain by using discrete cosine transform to obtain a frequency domain feature map; filter out high-frequency domain features for distinguishing the fine-grained features of the input image from the frequency domain feature map by using an adaptive threshold; convert the high-frequency domain features back to the spatial domain by using discrete cosine transform, and superimpose the converted spatial domain features with the spatial domain feature map output by the feature pyramid network to output a feature map after detail enhancement; The local feature refinement submodule is used to construct a channel attention mechanism and a spatial attention mechanism, use the channel attention mechanism to assign different weights to each channel, use the spatial attention mechanism to assign different weights to each spatial position, and output a feature map after local feature refinement based on the channel-level attention map calculated by the channel attention mechanism, the spatial-level attention map calculated by the spatial attention mechanism, and the feature map after detail enhancement output by the detail enhancement submodule; The adaptive feature fusion module is used to obtain information from the multi-scale feature map output by the backbone network, the feature map after detail enhancement output by the detail enhancement submodule, and the feature map after local feature refinement output by the local feature refinement submodule, dynamically adjust the fusion weight of each feature map through an adaptive weight learning mechanism, and output a fused feature; The fully connected layer is used to output a prediction result according to the fusion features output by the adaptive feature fusion module.

2. The Transformer-based fine-grained image classification system according to claim 1, characterized in that: The detail enhancement submodule specifically uses the following calculation formula to output the feature map after detail enhancement: in, is the feature map after detail enhancement, It is the spatial domain feature map output by the feature pyramid network. is the adaptive threshold function, is a parameter to adjust the enhancement amplitude. is the frequency domain feature map obtained by discrete cosine transform, is the inverse discrete cosine transform operation.

3. The Transformer-based fine-grained image classification system according to claim 1, characterized in that: The local feature refinement submodule specifically uses the following calculation formula to output the feature map after local feature refinement: in, is the feature map after local feature refinement, is the feature map after detail enhancement, is a channel-level attention map calculated by the channel attention mechanism, is the spatial level attention map calculated by the spatial attention mechanism, represents the dot product operation, Represents element-wise multiplication.

4. The Transformer-based fine-grained image classification system according to claim 1, characterized in that: The adaptive feature fusion module specifically uses the following calculation formula to output fusion features: in, To fusion features, Represents the activation function, specifically the ReLU function, Indicates a connection operation. is the weight obtained by adaptive learning, is the multi-scale feature map output by the backbone network, is the feature map after detail enhancement, It is the feature map after local feature refinement.

5. The Transformer-based fine-grained image classification system according to claim 1, characterized in that: The loss function of the system is a hybrid loss function formed by combining weighted cross entropy loss and region-sensitive loss; the weighted cross entropy loss is used to assign high weights to false sample categories and low weights to true sample categories; the region-sensitive loss is used to assign high weights to key areas and low weights to non-key areas.

6. The Transformer-based fine-grained image classification system according to claim 5, characterized in that: The calculation expression of the mixed loss function is: in, is the mixed loss function, and is the weight coefficient, is the weighted cross entropy loss, It is the area sensitive loss.

7. The Transformer-based fine-grained image classification system according to claim 6, characterized in that: The calculation expression of the weighted cross entropy loss is: in, is the weighted cross entropy loss, For category The weight of is the number of samples, For sample Belongs to category The real label, For the model category The predicted probability of .

8. The Transformer-based fine-grained image classification system according to claim 6, characterized in that: The calculation expression of the regional sensitivity loss is: in, is the regional sensitivity loss, It is a key area The weight of is an exponential function, which means that the sample Is it a region? , is the number of samples, is the basic loss function, is the number of critical regions, is the model’s predicted label for the sample, is the true label of the sample.

9. A Transformer-based fine-grained image classification method, characterized in that: Applied to the Transformer-based fine-grained image classification system according to any one of claims 1 to 8, the method comprising: Using the backbone network to process multi-scale information in an input image through a built-in moving window mechanism, and outputting a multi-scale feature map of the input image; The feature pyramid network is used to construct a feature pyramid to perform multi-scale feature extraction and fusion on the multi-scale feature map output by the backbone network, and output a spatial domain feature map; The detail enhancement submodule uses discrete cosine transform to transform the spatial domain feature map output by the feature pyramid network into the frequency domain to obtain a frequency domain feature map; a high-frequency domain feature for distinguishing the fine-grained features of the input image is selected from the frequency domain feature map by an adaptive threshold; the high-frequency domain feature is transformed back to the spatial domain by discrete cosine transform, and the transformed spatial domain feature is superimposed on the spatial domain feature map output by the feature pyramid network to output a feature map after detail enhancement; The local feature refinement submodule is used to construct a channel attention mechanism and a spatial attention mechanism, the channel attention mechanism is used to assign different weights to each channel, and the spatial attention mechanism is used to assign different weights to each spatial position, and based on the channel-level attention map calculated by the channel attention mechanism, the spatial-level attention map calculated by the spatial attention mechanism, and the feature map after detail enhancement output by the detail enhancement submodule, a feature map after local feature refinement is output; The adaptive feature fusion module is used to obtain information from the multi-scale feature map output by the backbone network, the feature map after detail enhancement output by the detail enhancement submodule, and the feature map after local feature refinement output by the local feature refinement submodule, and the fusion weight of each feature map is dynamically adjusted through an adaptive weight learning mechanism to output a fused feature; The fully connected layer is used to output a prediction result according to the fusion features output by the adaptive feature fusion module.

10. A storage medium storing a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to claim 9 are implemented.

Images