A self-supervised learning fine-grained image classification method based on Siamese network

Through a self-supervised learning method based on twin networks, the attention encoder and semantic erase mechanism are used to obtain different perspective features, and combined with the center and consistency loss function, the problem of feature consistency and accurate positioning in fine-grained image classification is solved, significantly improving the problem of feature consistency and accurate positioning in fine-grained image classification. Classification accuracy and model stability are obtained.

CN114676777BActive Publication Date: 2024-11-01INST OF SOFTWARE - CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210306415.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-25
Publication Date
2024-11-01
Estimated Expiration
2042-03-25

AI Technical Summary

Technical Problem

The fine-grained image classification task is difficult to effectively utilize image-level annotation information, which makes it difficult for the network to optimize loss functions and accurately locate semantic components, and existing methods cannot effectively capture the discernible features in the subtle areas.

Method used

Using a self-supervised learning method based on twin networks, image features are extracted through attention encoder, positioning and cropping significant areas, and viewing from different perspectives is formed, and the consistency of viewing angle features is constrained through self-supervised learning and loss function, combining with the central loss function. and consistency loss function, feature representation within significant regions.

Benefits of technology

It significantly improves the accuracy of fine-grained image classification and the robustness of the model, enhances the ability to identify subtle area features, and improves the generalization ability of the classification model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114676777B_ABST
    Figure CN114676777B_ABST
Patent Text Reader

Abstract

The present invention discloses a self-supervised learning fine-grained image classification method based on a twin network. The present invention first uses an attention encoder to extract deep convolutional features of an image, obtains an attention map containing semantic information, and encodes image features through a bilinear pooling method; secondly, the salient areas where high response values ​​are located on the attention map are located from the original image, and the salient areas are cropped and erased to form views from different perspectives, and learn perspective invariance features in a self-supervised manner; finally, the center loss function and the consistency loss function are combined to explicitly constrain different perspective features and maintain their intra-class consistency. The method of the present invention enables the network to obtain significant performance gains, and can significantly improve the classification accuracy on the baseline of fine-grained image classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to fine-grained image classification, and in particular to a self-supervised learning fine-grained image classification method based on a twin network. Background Art

[0002] With the increasing maturity of deep learning technology and the continuous improvement of social intelligence, people have put forward higher requirements for the classification ability of computers. It is expected that computers can not only correctly identify the targets of general image classification with large appearance differences, but also be able to identify objects at a fine-grained level. Fine-grained image classification refers to distinguishing subcategories belonging to the same basic category. It has a wide range of applications in vehicle tracking, automatic settlement of goods, protection of wild animals and plants, etc. Since obtaining component and object annotation information requires "expert-level" knowledge, the construction of fine-grained datasets consumes a lot of manpower and material resources. Therefore, the current research focus is on weakly supervised classification methods that only use image-level label information. Since subcategories belong to the same basic category, the appearance between subcategories is extremely similar, but due to external environmental factors such as background, lighting, and posture, the appearance of objects within a subcategory is quite different. The difficulty of this task lies in the small inter-class differences and intra-class similarities. In order to solve this problem, a weakly supervised method based on the attention mechanism is currently used to locate the distinguishable area, and the regional features and the overall image features are combined to complete the classification task. However, it is necessary to train multiple sub-networks / models, one for locating the semantic component and the other for the final classification prediction. This multi-task mode makes it difficult to optimize the network's loss function to an optimal situation. Furthermore, since there is only image-level annotation information, the network cannot accurately locate the area where the semantic component is located, and the acquired area contains more or less background noise. In contrast, although the method of encoding high-order features can enhance feature representation, it cannot explicitly guide the model to focus on the discernible features in subtle areas. Summary of the invention

[0003] In order to overcome the difficulties of fine-grained image classification, the purpose of the present invention is to provide a self-supervised learning fine-grained image classification method based on a twin network, as well as an electronic device and a scale storage medium. The present invention first uses an attention encoder to extract the deep convolution features of the image, obtains an attention map containing semantic information, and encodes the image features through a bilinear pooling method; secondly, the salient areas where the high response values ​​are located on the attention map are located from the original image, and the salient areas are cropped and erased to form views of different perspectives, and learn perspective invariant features in a self-supervised manner; finally, the center loss function and the consistency loss function are combined to explicitly constrain the features of different perspectives and maintain their intra-class consistency. The method of the present invention enables the network to obtain significant performance gains, thereby completing the present invention.

[0004] In order to achieve the purpose of the present invention, the present invention adopts the following steps:

[0005] 1) Use the backbone network in the attention encoder to extract the feature map F of the sample image I;

[0006] 2) Capture the high-order statistical information of the feature map F through the bilinear pooling module in the attention encoder to obtain the feature vector f(r) of the enhanced image;

[0007] 3) Based on the attention mechanism, locate the semantic components of the sample image I containing key information and obtain a cropped view sample I of the sample image I crop ;

[0008] 4) Based on the semantic erasing mechanism, another perspective sample of sample image I is obtained: Erased perspective sample I erase , thereby guiding the network to focus on the subtle but recognizable semantic components;

[0009] 5) The parameters of the twin network architecture are shared to extract the features of the two perspectives, namely f c and f e ;

[0010] 6) Through self-supervised learning, different perspectives of image I crop ,I erase As a positive sample pair, using the self-supervised learning loss function L ssl Constrain two view features f c and f e The consistency of ,captures the perspective invariant features;

[0011] 7) Use the center loss function to reduce the feature vectors I and I of the samples learned by the network crop and I erase The distance from the central feature C of the corresponding class compacts the sample features within the class and increases the stability of the model;

[0012] 8) Use the consistency loss function to constrain the similarity between the fully connected layer weights and the category center feature vector to maintain the different perspective features f c and f e The intra-class consistency also enables the fully connected layer to better integrate the sample features f(r) and f c and f e Mapping to category space;

[0013] 9) The fully connected layer combines the original sample image I and the features f(r), f c and f e Uniformly map to the final category probability distribution, using the cross entropy loss function L in the classifier clsCalculate the difference between the category probability distribution of sample image I and the unique hot encoding of the corresponding true label, reduce the difference, and optimize the overall network parameters.

[0014] Classification model application stage

[0015] The target image to be classified is input into the trained classification model to obtain the category of the target image.

[0016] The beneficial effects of the present invention include:

[0017] 1) Adopting twin networks and self-supervision to learn the view-invariant features of images at different viewpoints, and maintaining the intra-class consistency of features of samples at different viewpoints;

[0018] 2) The center loss function and consistency loss function further constrain the network to learn the intra-class consistency of the representation of similar samples, enhancing the robustness and generalization of the model;

[0019] 3) The self-supervised learning fine-grained image classification method based on the twin network provided by the present invention can significantly improve the classification accuracy on the baseline of fine-grained image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is the process of the self-supervised learning fine-grained image classification method based on the twin network described in an embodiment of the present invention.

[0021] Figure 2 This is the framework of the self-supervised learning fine-grained image classification method based on the twin network of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0023] Example 1

[0024] The self-supervised learning fine-grained image classification method based on the twin network of the present invention comprises the following steps:

[0025] Step S1: Construct an attention encoder. First, use the backbone network pre-trained on a large-scale classification dataset to extract the input image feature F; the backbone network has a wide range of options, and can choose residual networks (ResNet) and classic deep learning models VGG16. The backbone network is pre-trained on a large-scale classification dataset (such as ImageNet or Microsoft COCO). Second, capture rich image information through a bilinear pooling module. First, process the image feature F through a 1×1 convolution layer (Conv), an activation function (ReLU) and batch normalization (BN), thereby reducing the number of channels, discarding redundant information, and obtaining an attention map A with rich semantic information. Then, the image feature F and the attention map A are multiplied pixel by pixel to complete the semantic information, and then a global average pooling layer is passed to obtain the final feature vector f(r) of the image.

[0026] Step S2: Acquire images at different viewing angles: cropping viewing angle and erasing viewing angle.

[0027] Cropping perspective: First, randomly select M feature maps from A to form A′, as shown below. First, convert the M feature maps into probability distribution a i ,

[0028]

[0029] Then, a weighted summation method is adopted to obtain the feature map D containing complete semantic information, as shown below.

[0030]

[0031] Next, the crop mask M of the image is obtained through the feature map D D , as shown below.

[0032]

[0033] Finally, using the original image I and the crop mask image M D By multiplying each pixel point by point, the high response area corresponding to the feature map D is cut out from the original image to generate the cut view angle I crop .

[0034] I crop =I⊙M D

[0035] Erasing perspective: In order to enhance the robustness of the model, a random attention map A″ is selected from the attention map, and the pixel values ​​of the high response area on it are assigned to 0, thereby erasing the high response area on A″ to generate an erasure mask map M e , guiding the network to focus on the remaining distinguishable regions, as shown below.

[0036]

[0037] Finally, the sample image I of another perspective is obtained by point-by-point multiplication. erase .

[0038] I erase =I⊙M e

[0039] Step S3: Extract features f of two perspectives through the Siamese network c and f e , using self-supervised learning to maintain the consistency of features from different perspectives. The images from the two perspectives generated in step S2 are input into the parameter-sharing twin network architecture to obtain the features f of the two perspectives. c and f e , the two perspective features are constrained through self-supervised learning, as shown below.

[0040] L ssl =||f c -f e ||2

[0041] Step S4: Construct sample category centers. First, for each category included in the data set, a central feature vector C is established, which is initialized to obey the normal distribution; then, the central loss function L is used to ct Constrain the distance between each sample feature and its class center feature vector. The specific calculation process of the center loss function is as follows, where f∈(f(r), f c , f e ).

[0042] L ct =||fC||2

[0043] During the training process, the training process is iterated continuously to obtain a representative class center feature vector. The specific process is as follows, where C and C′ represent the true class center corresponding to f(r) before and after the iteration, and α is the step size for updating the class center.

[0044] C′=C+α(f(r)-C)

[0045] Step S5: Using the consistency loss function L cst , further constraining the intra-class consistency of different perspectives, as shown below. total It is composed of the class center feature C corresponding to each category of the data set, and W is the weight of the fully connected layer, which can also represent the distribution of class center features.

[0046] L cst =||C total -W||2

[0047] Minimize the difference between the category center shared by samples from different perspectives and the weight of the fully connected layer to ensure that the fully connected layer maps the features of samples from different perspectives to the label space of the samples of their respective categories.

[0048] Step S6: Output the fine-grained image category predicted by the network through the classifier.

[0049] Step S7: Train the entire network using a stochastic gradient descent algorithm. When the preset number of rounds is reached, stop training the entire network.

[0050] Step S8: Test the network. Send the target image to the trained network model, extract the feature representation of the target image through the attention encoder, and send it to the classifier to obtain the probability value of the classifier classifying the target image into each category. The category with the highest probability value is the final category label of the target image.

[0051] Example 2

[0052] Embodiment 2 of the present invention provides an electronic device, which includes a memory and a processor, wherein when a self-supervised learning fine-grained image classification program based on a twin network is stored and executed by the processor, the processor executes a self-supervised learning fine-grained image classification method based on a twin network, and the method includes the following steps:

[0053] 1) Use the constructed attention encoder to extract feature representation of the input image;

[0054] 2) Obtain two different perspective samples of the image based on attention and semantic erasure mechanism;

[0055] 3) Through the twin network, extract the features of samples from different perspectives and guide the network to learn the perspective invariant features in a self-supervised manner;

[0056] 4) Use the center loss function to constrain the feature distance of samples within the class;

[0057] 5) The samples learned by the network and their features from different perspectives are sent to the classifier to obtain fine-grained image categories.

[0058] Example 3

[0059] Embodiment 3 of the present invention provides a computer-readable storage medium, characterized in that when the program is executed by a processor, the processor executes a self-supervised learning fine-grained image classification method based on a twin network, the method comprising the following steps:

[0060] 1) Use the constructed attention encoder to extract the feature representation of the input image;

[0061] 2) Use the attention mechanism and semantic erasure mechanism to obtain two different perspective samples of the image;

[0062] 3) Through the twin network, extract the features of samples from different perspectives and guide the network to learn the perspective invariant features in a self-supervised manner;

[0063] 4) Use the center loss function to constrain the feature distance of samples within the class;

[0064] 5) The samples learned by the network and their features from different perspectives are sent to the classifier to obtain fine-grained image categories.

[0065] The above description is only a preferred example of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A self-supervised learning fine-grained image classification method based on a twin network, the steps of which include: In the classification model training stage, the classification model is trained based on the training sample image set using steps 1) to 9); wherein the classification model includes a first attention encoder, a second attention encoder, a fully connected layer and a classifier; 1) Use the first attention encoder to extract the feature map F of the sample image I; 2) The first attention encoder generates a feature vector f(r) of the enhanced image according to the high-order statistical information of the feature map F; 3) The first attention encoder obtains the cropped view sample I of the sample image I by locating the semantic components containing key information in the sample image I crop ; 4) The first attention encoder obtains the erased view sample I of the sample image I based on the semantic erasure mechanism erase ; 5) Cut the view sample I crop , Erase View Sample I erase Input the second attention encoder respectively to obtain the corresponding cropping perspective feature vector f c and erase the view feature vector f e ; The second attention encoder is a parameter-sharing twin network architecture; 6) Cut the view sample I crop , Erase View Sample I erase As a positive sample pair, using the self-supervised learning loss function L ssl Constrain the two view feature vectors f c and f e The consistency of the image is captured by self-supervised learning. 7) Using the center loss function L ct , reduce the distance between the view invariant feature of the sample image I and the class center feature C of the class to which the sample image I belongs; 8) The eigenvectors f(r) and f c and f e Input the fully connected layer to get f(r), f c and f e Unify to the same category probability; use the consistency loss function L cst Constrain the similarity between the fully connected layer weights and the category center features; 9) The fully connected layer is based on the sample image I, f(r), f c and f e Get the category probability distribution of sample image I; then use the cross entropy loss function L in the classifier cls Calculate the difference between the category probability distribution of the sample image I and the one-hot encoding of the corresponding true label, and optimize the weight of the fully connected layer by reducing the difference; Classification model application stage The target image to be classified is input into the trained classification model to obtain the category of the target image.

2. The method according to claim 1, characterized in that The center loss function is L ct =‖fC‖2; where f∈(f(r),f c ,f e ).

3. The method according to claim 2, characterized in that The consistency loss function is L cst =‖C total -W‖2; Among them, C total It is composed of the class center features C corresponding to each category of the training sample image set, and W is the weight of the fully connected layer.

4. The method according to claim 1, 2 or 3, characterized in that: The self-supervised learning loss function is L ssl =‖f c -f e ‖2.

5. The method according to claim 1, 2 or 3, characterized in that: The training process updates the class center feature through C′=C+α(f(r)-C); where C′ represents the class center feature after iteration and α is the step size for updating the class center.

6. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.