A fine-grained image classification method based on Transformer
By introducing a dual-branch architecture and a central loss function in Transformer, the problems of small and excessive smoothing of feature differences in fine-grained image classification are solved, and the classification accuracy is significantly improved, especially in the CUB-200-2011, NABirds and iNat2017 datasets performance on.
Patent Information
- Application Number
- CN202210305985.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-25
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-03-25
AI Technical Summary
In the fine-grained image classification task, the existing Transformer has problems such as poor detailed ability of learning objects, excessive smooth features, insufficient fusion of multi-grained features, and small feature differences between subcategories, resulting in limited classification accuracy.
Using a dual-branch architecture, multi-level features and local details are extracted through the token feature selection module and semantic component generation module, and the central loss function constrains the feature distance within the class to enhance the model's ability to capture subtle differences.
It significantly improves the accuracy of fine-grained image classification, especially the best results were achieved on the CUB-200-2011, NABirds and iNat2017 datasets, and enhances the model's ability to recognize subtle features.
Smart Images

Figure CN114676776B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image classification, and in particular to fine-grained image classification, and in particular to a Transformer-based fine-grained image classification method. Background Art
[0002] Fine-grained image classification is a practical and challenging computer vision task, which aims to identify different subcategories under the same category. In recent years, with the rapid development of deep learning and artificial intelligence technology, this task has been greatly promoted and has been widely used in fields such as virus identification, commodity retail, inventory management, and autonomous driving. Many works use convolutional neural networks to learn distinguishable features in images, but convolutional neural networks are limited by the size of the convolution kernel and cannot construct the semantic information of the entire image. Therefore, the accuracy improvement of fine-grained image classification tasks has encountered a bottleneck. The birth of visual Transformer has completely broken this limitation and greatly improved the classification accuracy of this task. It serializes images into tokens similar to those in natural language processing tasks, and completes various computer vision tasks by building relationships between tokens. Unlike convolutional neural networks, which can only build relationships between pixels in adjacent areas, visual Transformer has a more robust ability to build global representations of images through graph modeling mechanisms, but there are still some problems in migrating Transformer to fine-grained image classification tasks. First, the Transformer has a strong global modeling capability, which inevitably leads to its poor ability to learn object details, and distinguishable detail features are crucial for correctly identifying fine-grained image categories. Second, as the number of Transformer structure layers increases, the features extracted by the entire network are overly smoothed, resulting in the network's inability to capture diverse discernible features. Third, ignoring the fusion of multi-granular features extracted by the network leads to a loss in model performance. Fourth, the difficulty of fine-grained image classification is not taken into account, that is, the appearance of objects between subcategories is extremely similar, and the feature differences are small; due to external environmental factors such as lighting, posture, and occlusion, the appearance of objects within a subcategory is very different, and the feature similarity is small. Summary of the invention
[0003] In order to overcome the above problems, the purpose of the present invention is to provide a Transformer-based fine-grained image classification method, an electronic device and a storage medium. First, the image block token features containing the most information are selected from the encoder layer of the Transformer and combined into a multi-level image feature representation from low to high levels. Secondly, a dual-branch architecture is adopted to explicitly learn distinguishable features in fine-grained areas and enhance the model's ability to capture subtle differences. Third, the classification tokens of the two branches are exchanged to fuse global and local information. Fourth, the sample feature distance is constrained by the central loss function, thereby alleviating the difficulty of large feature gaps within the class and small feature differences between classes. Our method enables the network to obtain significant performance gains, thereby completing the present invention.
[0004] In order to achieve the purpose of the present invention, the present invention adopts the following steps:
[0005] 1) Input the sample image into the linear mapping and encoder layer to extract multi-layer token features {Z1,…,Zn-2};
[0006] 2) The token feature selection module acts on the multi-layer token features {Z1,…,Zn-2} extracted by the encoder to obtain multi-layer features with more effective discrimination for classification results;
[0007] 3) The semantic component generation module acts on the first-layer token feature Z1 to obtain the local area containing the recognizable visual pattern, and the local branch is used to learn the detailed information of the object in the area;
[0008] 4) The classification tokens of the two branches are exchanged, and the encoder acts on the exchanged classification tokens and the image block tokens of the current branch, which simply and effectively fuses the complete image information extracted by the global branch and the detailed information of the key area obtained by the local branch;
[0009] 5) Through the center loss function, the feature distance within the fine-grained object class is further constrained, and the feature distance between classes is indirectly increased;
[0010] 6) Connect the classification tokens of the two branches and input them into the classifier to achieve classification of the input image.
[0011] A server includes a memory and a processor, wherein a computer program is stored in the memory and the processor executes the computer program, wherein the computer program includes instructions for executing each step in the above method.
[0012] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method when executed by a processor.
[0013] The beneficial effects of the present invention include:
[0014] 1) This paper proposes a new dual-branch Transformer model, which includes three modules: token feature selection, semantic component generation and feature fusion module;
[0015] 2) The Transformer-based fine-grained image classification method provided by the present invention can achieve significant performance improvement on the baseline of fine-grained image classification algorithms. In particular, the best classification results were achieved in three benchmark datasets (CUB-200-2011, NABirds, and iNat2017). BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 This is the process of the Transformer-based fine-grained image classification method described in an embodiment of the present invention;
[0017] Figure 2 The framework of the Transformer-based fine-grained image classification method described in the present invention is shown;
[0018] Figure 3 shows a block diagram of the encoder. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0020] Example 1
[0021] The Transformer-based fine-grained image classification method of the present invention comprises the following steps:
[0022] Step S1: Construct linear mapping and encoder layers and pre-train them on the large-scale classification dataset ImageNet to extract multi-level token features {Z1,…,Z n-2}, the specific process is as follows:
[0023] like Figure 2 As shown, first, the linear mapping divides the input image into N image blocks x p , through the hidden space E, the image block x p Convert to image block token x p E. Next, manually add the image block token x p E Classification tokens with the same feature dimension The classification token is the most important token in the Transformer structure. The classifier classifies the classification token to complete the classification of the input image. Finally, because the image is serialized into a one-dimensional sequence, the pixels in the image lose their spatial position information. Therefore, an additional position feature vector E is added. pos , thereby preserving the position information between image blocks. The token features of the input encoder layer 1 are as follows: Corresponding classification tokens, j=1, 2…N correspond to N image block tokens respectively.
[0024]
[0025] Next, through Figure 3 The encoder layer shown in the figure shows the effective information between the interactive tokens. The encoder structure of each layer in the Transformer structure is the same. Each encoder layer consists of a multi-head attention mechanism and a multi-layer perceptron. The information interaction process is as follows: The input data of the encoder layer w (w∈[1,n-2]) is the output Z of the encoder layer (w-1) w-1 , the output data is Z w The multi-head attention mechanism (MSA) acts on the token features Z after layer normalization (LN) w-1 , to achieve information interaction between tokens, residual connection acts on Z w-1 and MSA(LN(Z w-1 )). Multilayer Perceptron (MLP) acts on Z′ after LN processing. w , the residual connection acts on Z′ w and MLP(LN(Z′ w )).
[0026] Z′ w =MSA(LN(Z w-1 ))+Z w-1
[0027] Z w =MLP(LN(Z′ w ))+Z′ w
[0028] Step S2: Construct a token feature selection module. First, the multi-head attention mechanism divides the token features into K subspaces. In each subspace, the interaction information between tokens is modeled so that the feature representation of each token has K different spatial distributions. Then, in each subspace, the image block token feature with the highest degree of correlation with the classification token is selected, so that the token features of each layer from the 1st layer to the n-2th layer can be obtained. (w∈[1,n-2]). It is worth noting that the image patch token with the highest correlation with the classification token in different subspaces is not the same. For example, in the i-th subspace, the m-th image patch token is most similar to the classification token, while in the j-th subspace, the n-th image patch token is most similar to the classification token. Finally, the token features selected from the 1st to the n-2nd layers are It is concatenated with the classification token of the n-2 layer, and the multi-level information carried by the selected key features is fused to the classification token using the encoder layer n-1.
[0029] Step S3: Construct a semantic component generation module. For the first-layer token feature Z1 obtained in step S1, select the image block token whose correlation with the classification token is greater than the threshold θ according to the correlation with the classification token. The threshold θ is an empirical value selected after multiple experiments. Calculate the coordinates of the four vertices of the rectangular area containing all the selected image block tokens. Then cut the rectangle from the input image to obtain the local area containing the semantic component.
[0030] Step S4: The local branch explicitly captures the subtle differences in semantic components. The principle is that the input of this branch is a local area that only contains semantic components. Therefore, compared with the global branch, the local branch can focus on the details with differences in semantic components. The local branch structure is the same as the global branch structure, and both are composed of linear mapping, encoding layer, and token feature selection module. Although the two structures are the same, the parameters are not shared, so as to ensure that the two branches can focus on extracting feature representations of corresponding scales.
[0031] Step S5: Construct a feature fusion module. First, directly exchange the classification tokens in the token features output by the penultimate layer of the two visual Transformer branches. Next, concatenate the classification tokens with the image block token features in the token features output by the penultimate layer of the current branch, and exchange the classification token and image block token information through the encoder layer n. Finally, concatenate the global classification tokens with the local classification tokens, and process the obtained features through the fully connected layer in the classifier, so that the two classification tokens contain richer information that is helpful for classification.
[0032] Step S6: Use the center loss function to compact the feature distance within the fine-grained object class and expand the feature distance between classes. The center loss function formula is as follows. Where x corresponds to the concatenated global classification token and local classification token, and the center feature vector c is a vector with the same feature dimension as x, and its initialization follows a normal distribution. Through the center loss function, during the training iteration process, the center feature vector c is continuously updated and the sample feature x is constrained to be close to the center feature vector.
[0033] L ct =‖xc‖2
[0034] Step S7: Train the network using a gradient descent algorithm. When the number of rounds specified in advance is reached, the entire network stops training.
[0035] Step S8: Network test: The image to be classified is sent to the trained network model to obtain the probability value of the classifier classifying the target image into each category. The category with the highest probability value is the final category label of the target image.
[0036] Example 2
[0037] Embodiment 2 of the present invention provides an electronic device, including a memory and a processor, wherein when a Transformer-based fine-grained image classification program is stored and executed by the processor, the processor executes a Transformer-based fine-grained image classification method, the method comprising the following steps:
[0038] 1) Use the pre-trained encoder layer to extract multi-level features of the input image;
[0039] 2) Using the token feature selection module to select the image block token features containing the most information from each layer of feature representation;
[0040] 3) Locate the semantic components and send them to the local branch to extract the information in the area that is effective for classification;
[0041] 4) Replace the classification tokens of the two branches to aggregate global and local information;
[0042] 5) Use the center loss function to reduce the feature distance of samples within a class and increase the feature distance of samples between classes. Finally, the classifier acts on the classification token and outputs the predicted category probability. The category with the maximum probability value is used to predict the category of the fine-grained image for the network.
[0043] Example 3
[0044] Embodiment 3 of the present invention provides a computer-readable storage medium, characterized in that when the program is executed by a processor, the processor executes a Transformer-based fine-grained image classification method, the method comprising the following steps:
[0045] 1) Use the pre-trained encoder layer to extract multi-level features of the input image;
[0046] 2) Using the token feature selection module to select the image block token features containing the most information from each layer of feature representation;
[0047] 3) Locate the semantic components and send them to the local branch to extract the information in the area that is effective for classification;
[0048] 4) Replace the classification tokens of the two branches to aggregate global and local information;
[0049] 5) The center loss function is used to reduce the feature distance of samples within a class and increase the feature distance of samples between classes. Finally, the classifier outputs the predicted category of the fine-grained image.
[0050] The above description is only a preferred example of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A Transformer-based fine-grained image classification method, the steps of which are include: Constructing an image classification model, including a local branch, a global branch, a semantic component generation module, a feature fusion module and a classifier, wherein the local branch and the global branch both include a linear mapping and encoder layer, and a token feature selection module; Train the image classification model: The linear mapping and encoder layers extract features from the input sample image to obtain multi-layer token features {Z 1 ,…,Z n-2 }; The linear mapping and encoder layer includes a linear mapping unit and n encoders that process the input data in sequence, Z n-2 The feature output by the n-2th encoder, that is, the n-2th layer token feature; The token feature selection module selects the token features from the multi-layer token features {Z 1 ,…,Z n-2 }Select multi-level features with discriminativeness; The semantic component generation module generates the first-layer token features Z from the output of the first encoder. 1 Acquire a local region containing a discernible visual pattern, and then use the local branch to learn detailed information of the object in the local region; The feature fusion module exchanges the classification token in the token feature output by the n-1th encoder in the local branch with the classification token in the token feature output by the n-1th encoder in the global branch; then the local branch splices the exchanged classification token with the image block token feature in the token feature output by the n-1th encoder in the local branch and inputs the spliced image block token feature in the token feature output by the n-1th encoder in the global branch and inputs the spliced image block token feature in the token feature output by the n-1th encoder in the global branch and inputs the spliced image block token feature in the token feature output by the n-1th encoder in the global branch and inputs the spliced image block token feature in the token feature output by the n-1th encoder in the global branch and inputs the spliced image block token feature in the classifier to obtain the category of the input sample image; Constraining the feature distance within the object class in the sample image through a set loss function; Image classification model application stage: The target image to be classified is input into the trained image classification model to obtain the category label of the target image.
2. The method according to claim 1, It is characterized in that The linear mapping unit first divides the input sample image into N image blocks x p , through the hidden space E, each image block x p Convert to an image block token x p E; then add the image block token x p E categorical tokens with the same feature dimension Get the serialization information corresponding to the sample image And input to the encoder layer; where E pos is a position feature vector for retaining the position information between the image blocks of the sample image, is the jth image block x p The corresponding image block token, j=1,2…N.
3. The method according to claim 2, It is characterized in that The encoder is composed of a multi-head attention mechanism MSA and a multi-layer perceptron MLP. The multi-head attention mechanism of the w-th layer encoder firstly performs the token feature Z output by the (w-1)-th layer encoder. w-1 Perform layer normalization LN, and the obtained data is recorded as MSA(LN(Z w-1 )); then Z w-1 and MSA(LN(Z w-1 )) Input the first residual connection unit to obtain feature Z′ w ; Multilayer Perceptron for Z′ w Perform layer normalization LN, and the obtained data is recorded as MLP(LN(Z′ w )), Z′ w and MLP(LN(Z′ w )) Input the second residual connection unit to obtain the w-th layer token feature Z w ; w∈[1,n-2].
4. The method according to claim 3, It is characterized in that The token feature selection module first uses a multi-head attention mechanism to divide each layer of token features into K subspaces. In each subspace, the interaction information between tokens is modeled so that each token feature representation has K different spatial distributions. Then, in each subspace, the image block token feature with the highest degree of correlation with the classification token is selected as the token feature of the corresponding layer. Set the selected token features As a discriminative multi-level feature.
5. The method according to claim 1, It is characterized in that The loss function is the center loss function L ct =‖xc‖ 2 ; Where c is the central feature vector, and x is the concatenated global classification token feature and local classification token feature.
6. The method according to claim 1, It is characterized in that The semantic component generation module selects token features Z 1 The image block tokens whose correlation degree with the classification token is greater than the set threshold θ are calculated, and the coordinates of the four vertices of the rectangular area containing all the selected image block tokens are calculated; then the rectangular area determined by the four vertices is cut out from the input image as the local area.
7. A server, It is characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method according to any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Face anti-counterfeiting model training method and device and electronic equipment
CN113553904A
Pollen image classification method based on cross attention distillation Transformer
CN113887610A
Taking identification method and device based on double-channel cross attention mechanism
CN113936339A
Image semantic segmentation method based on cascaded multi-scale visual Transformer
CN113947680A
Image classification method and device, electronic device and storage medium
CN114155395A
