A fine-grained image classification method based on visual feature constraints

Abstract

The present invention discloses a fine-grained image classification method based on visual feature constraints. The method includes the following steps: using a pre-trained visual feature encoder of the CLIP method to extract intermediate features of an image to constrain the intermediate features extracted by a fine-grained image classification model; using text features extracted by a pre-trained text feature encoder of the CLIP method to supervise and constrain the extracted image features for fine-grained image classification; using the pre-trained visual feature encoder of the CLIP method to obtain an activation map of a training image, masking the training image with the activation map as a mask, and then obtaining visual features of the masked image; combining the masked image features with ordinary visual features before classification. This method utilizes the image and text feature encoders of the CLIP method to help the fine-grained image classification model better extract the visual features of the fine-grained image, thereby helping to improve the classification accuracy of the fine-grained image classification model.

Description

Technical Field

[0001] The present invention belongs to computer vision technology, and in particular relates to a fine-grained image classification method based on visual feature constraints. Background Art

[0002] In natural language processing, pre-trained models obtained by using large amounts of unprocessed data as training data can be applied to different downstream tasks. However, in computer vision, a large number of pre-trained models can only be applied to some datasets with a distribution similar to that of the training dataset, and cannot be applied to multiple types of downstream tasks like natural language processing models. Therefore, researchers proposed the CLIP (Contrastive Language-Image Pre-training) method. This method makes full use of the large amount of paired text and image data that can be easily crawled and collected on the Internet, and uses text as image labels to train a model with strong generalization ability, which is easy to transfer to other downstream tasks.

[0003] The CLIP method's main process is as follows: First, 400 million images are obtained from search engines using 500,000 query texts. Then, a visual feature encoder and a text feature encoder are used to extract image and text features, respectively. Finally, the paired visual and text feature encoders are trained using metric learning. The CLIP method ultimately produces a visual feature encoder that extracts image features and a text feature encoder that extracts text features. The features extracted by the two encoders are in the same feature space, and similarity can be determined by comparison. Currently, many downstream tasks use the CLIP method to improve performance by leveraging pre-trained encoders, but the CLIP method has not yet been applied to fine-grained image classification. Fine-grained image classification datasets have a corresponding text description for each image, which can be combined with the text feature encoder trained using the CLIP method to extract text features to aid fine-grained image classification. By constraining the visual features extracted by the model, the accuracy of the fine-grained model can be improved. Summary of the Invention

[0004] (1) Technical issues to be solved

[0005] To solve the problem that many downstream tasks currently use the CLIP method to improve performance with the help of pre-trained encoders, but the CLIP method has not yet been applied to the field of fine-grained image classification, a fine-grained image classification method based on visual feature constraints is provided.

[0006] (2) Technical solution

[0007] The purpose of this invention is to use the visual feature encoder and text encoder obtained in the CLIP method to help improve the performance of the fine-grained image classification model, and provide a fine-grained image classification method based on visual feature constraints, which specifically includes the following steps:

[0008] Step 1: Collection of training data set;

[0009] Step 2: Perform data preprocessing and data augmentation on the training images;

[0010] Step 3: Use the ViT-B / 16 model in the CLIP method as the base model to extract the visual features of the training image;

[0011] Step 4: Use the ViT-B / 16 visual feature encoder pre-trained on a large dataset in the CLIP method to extract visual features of the image trained in step 3. Each layer of the encoder outputs the intermediate features of the extracted image. These intermediate features are used as standards to constrain the intermediate features of the ViT-B / 16 model in step 3, obtaining image feature constraint 1.

[0012] Step 5: Each training image has corresponding descriptive text data. The transformer text feature encoder pre-trained on a large dataset in the CLIP method is used to extract the descriptive text data corresponding to the training image in step 3 to obtain text features. These text features are used as standards to constrain the image visual feature vector obtained by the model in step 3, obtaining image feature constraint 2.

[0013] Step 6: Use the ViT-B / 16 visual feature encoder pre-trained on a large dataset in the CLIP method to obtain the activation map of the training image, and use the activation map as a mask to mask the training image;

[0014] Step 7: Use the model in step 3 to extract the visual features of the image after mask processing in step 6 to obtain the visual features of the masked image;

[0015] Step 8: The image features obtained by combining the common visual features obtained in steps 3 and 7 with the visual features of the masked image are used as the final image visual features of the training image; the final image visual features are passed through a multi-layer perceptron to obtain the confidence level of each class, and the classification loss is calculated using the cross entropy loss function;

[0016] Step 9: Add the image feature constraint 1, image feature constraint 2, and classification loss from steps 4, 5, and 8 to get the total loss for the task; use this total loss to train the ViT-B / 16 model from step 3.

[0017] Step 10: During the testing phase, make four copies of the test image, rotate each copy by a certain angle, and use the ViT-B / 16 model trained in step 9 to predict each of the four test images. The four output results are then averaged, and the category with the highest average score is the predicted category for the test image.

[0018] As a preferred technical solution, in step 1, the dataset used is the Caltech-UCSD Birds-200-2011 bird classification dataset, which contains image data of 200 types of birds, a total of 11,788 images, and each image has a corresponding descriptive text data; 5,994 images in this dataset are taken as the training dataset.

[0019] As a preferred technical solution, in step 2, the images are scaled to a uniform size of 224×224, and then data augmentation methods such as random cropping, random flipping, and random Gaussian blurring are used to increase the number of training images.

[0020] As a preferred technical solution, the input of the ViT-B / 16 model in step three is to cut the image into 16×16 blocks, and the output is a 768-dimensional visual feature vector of the image. Finally, a multi-layer perceptron is used to obtain the category score of the image.

[0021] As a preferred technical solution, in step 4, the ViT-B / 16 model has a total of 12 layers. The intermediate features output by the last 4, 3, and 2 layers of the pre-trained ViT-B / 16 model are used as the constraint features of image feature constraint 1. The calculation formula for image feature constraint 1 is as follows:

[0022] L1=Σ||f1-f2|| 2 ,

[0023] Where f1 is the intermediate feature extracted by the pre-trained model in step 4, and f2 is the corresponding intermediate feature of the model in step 3.

[0024] As a preferred technical solution, in step 5, the text feature dimension extracted by the transformer text feature encoder is 768-dimensional. This text feature is used to perform image feature constraint 2 on the image visual features extracted by the model in step 3. The calculation formula for image feature constraint 2 is as follows:

[0025] L2=||f t -f i || 2 ,

[0026] where f t is the text feature extracted by the text transformer feature encoder in step 5, f iIt is the visual feature of the image extracted by the model in step 3.

[0027] As a preferred technical solution, the pre-trained visual feature encoder used in step six obtains an activation map of the training image; the activation map is used to mask the training image; specifically, the pixel points at the relative positions of the training image with activation map values ​​greater than 0.5 are retained, and the pixel points at the relative positions of the training image with activation map values ​​less than 0.5 are set to zero.

[0028] As a preferred technical solution, the common visual features obtained in steps 3 and 7 are combined with the visual features of the mask image in step 8. The calculation formula of the combined features is as follows:

[0029] f=αf i +(1-α)f m ,

[0030] where f i is the visual feature of the last layer of the image extracted by the model in step 3, f m It is the visual feature of the mask image extracted in step 7, and α is a parameter that adjusts the importance of the two features.

[0031] As a preferred technical solution, the rotation angles in step ten are 0°, 90°, 180° and 270° respectively.

[0032] (3) Beneficial effects

[0033] The beneficial effects of the present invention are as follows: this method helps improve the performance of fine-grained image classification models by using the visual features of the pre-trained image visual feature encoder and the text feature encoder constraint model obtained by the CLIP method under a large amount of text-image pairing data. Specifically, the intermediate features of different layers of the pre-trained visual feature encoder are used as visual feature constraint 1 to help fine-grained image classification extract visual features; the text features extracted by the text feature encoder are used as visual feature constraint 2 to help fine-grained image classification better learn the details corresponding to the text in the image; the activation map obtained by the pre-trained visual feature encoder is used to mask the training image, and the visual features of the masked image are extracted and combined with the ordinary visual features before classifying the image. Using the activation map to mask the image can better extract the feature areas that the model focuses on and reduce the influence of background and other areas on feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 It is the overall model structure;

[0036] Figure 2 It is the process of obtaining mask image features;

[0037] Figure 3 It is the testing phase process. DETAILED DESCRIPTION

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] like Figure 1 As shown, the main steps of the technical solution adopted by the present invention are as follows:

[0040] Step 1: Use the Caltech-UCSD Birds-200-2011 bird classification dataset as the training dataset, which contains 200 bird categories and a total of 5994 training images.

[0041] Step 2: Scale the training images in step 1 to a uniform size of 224×224, and use data augmentation methods such as random cropping, random flipping, and random Gaussian blur to increase the number of training images.

[0042] Step 3: Use the ViT-B / 16 model input in the CLIP method to divide the image into 16×16 blocks, output 768-dimensional image visual features, and use the multi-layer perceptron to obtain the classification score for each category after extracting the visual features.

[0043] Step 4: Use the ViT-B / 16 visual feature encoder pre-trained on a large dataset in the CLIP method to extract the visual features of the training image in step 3. Each layer of the encoder outputs the extracted intermediate features. These intermediate features are used as standards to constrain the intermediate features of the ViT-B / 16 model in step 3, that is, Figure 1 Middle picture feature constraint 1.

[0044] Step 5: Each training image has corresponding descriptive text data. The transformer text feature encoder pre-trained on a large dataset in the CLIP method is used to extract the descriptive text data corresponding to the training image in step 3 to obtain text features. These text features are used as standards to constrain the final visual feature vector extracted in step 3, that is, Figure 1 Middle picture feature constraint 2.

[0045] Step 6: Use the ViT-B / 16 visual feature encoder pre-trained on a large dataset in the CLIP method to obtain the activation map of the training image, and use the activation map as a mask to mask the training image, such as Figure 2 As shown in .

[0046] Step 7: Use the model in step 3 to extract the visual features of the image after mask processing in step 6, and get Figure 2 Mask image features.

[0047] Step 8: Combine the common visual features obtained in Steps 3 and 6 with the mask image visual features to obtain the final visual features for the training image. Use α to adjust the importance of the two features. Testing shows that α = 0.5 works best. Pass the final visual features through a multi-layer perceptron to obtain the confidence level for each class, and then calculate the classification loss using cross-entropy loss.

[0048] Step 9: Add the image feature constraint 1, image feature constraint 2, and classification loss from steps 4, 5, and 8 to get the total loss. Use this total loss to train the ViT-B / 16 model from step 3.

[0049] Step 10: Testing phase, such as Figure 3 As shown in the figure, the test image is copied four times, each of which is rotated by a certain angle. The ViT-B / 16 model trained in step nine is then used to predict the four test images respectively. The output results of the four copies are then averaged. The category with the highest average score is the predicted category of the test image.

[0050] It should be noted that: the present invention uses the pre-trained visual feature encoder and text feature encoder obtained by the CLIP method under a large text image dataset to extract the visual features of the training image and the text features of the paired text to help the fine-grained image classification model better identify fine-grained categories. Specifically, first, the intermediate features obtained by extracting the image by the pre-trained visual feature encoder are used as the intermediate features extracted by the fine-grained classification model constrained by visual constraint 1. Secondly, the text feature supervision extracted by the pre-trained text feature encoder is used as the final image feature extracted by the fine-grained classification model constrained by visual feature constraint 2. Finally, the pre-trained visual feature encoder is used to obtain the activation map of the training image, which is used as a mask to mask the training image and then obtain the visual features of the masked image, which are combined with the ordinary visual features for classification.

[0051] The above embodiments are merely descriptions of preferred implementations of the present invention and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary persons in the art should fall within the scope of protection of the present invention. The technical contents for which protection is sought in the present invention are all recorded in the claims.

Claims

1. A fine-grained image classification method based on visual feature constraints, characterized by: The following steps are involved: Step 1: Collection of training data set; Step 2: Perform data preprocessing and data augmentation on the training images; Step 3: Use the ViT-B / 16 model in the CLIP method as the base model to extract the visual features of the training image; Step 4: Use the ViT-B / 16 visual feature encoder pre-trained on a large dataset in the CLIP method to extract visual features of the image trained in step 3. Each layer of the encoder outputs the intermediate features of the extracted image. These intermediate features are used as standards to constrain the intermediate features of the ViT-B / 16 model in step 3, obtaining image feature constraint 1. Step 5: Each training image has corresponding descriptive text data. The transformer text feature encoder pre-trained on a large dataset in the CLIP method is used to extract the descriptive text data corresponding to the training image in step 3 to obtain text features. These text features are used as standards to constrain the image visual feature vector obtained by the model in step 3, obtaining image feature constraint 2. Step 6: Use the ViT-B / 16 visual feature encoder pre-trained on a large dataset in the CLIP method to obtain the activation map of the training image, and use the activation map as a mask to mask the training image; Step 7: Use the model in step 3 to extract the visual features of the image after mask processing in step 6 to obtain the visual features of the masked image; Step 8: The image features obtained by combining the common visual features obtained in steps 3 and 7 with the visual features of the masked image are used as the final image visual features of the training image; the final image visual features are passed through a multi-layer perceptron to obtain the confidence level of each class, and the classification loss is calculated using the cross entropy loss function; Step 9: Add the image feature constraint 1, image feature constraint 2, and classification loss from steps 4, 5, and 8 to get the total loss for the task; use this total loss to train the ViT-B / 16 model from step 3. Step 10: During the testing phase, make four copies of the test image, rotate each copy by a certain angle, and use the ViT-B / 16 model trained in step 9 to predict each of the four test images. The four output results are then averaged, and the category with the highest average score is the predicted category for the test image.

2. A fine-grained image classification method based on visual feature constraints according to claim 1, characterized in that: In step 1, the dataset used is the Caltech-UCSD Birds-200-2011 bird classification dataset, which contains 11,788 images of 200 bird categories, and each image has a corresponding descriptive text data; 5,994 images in the dataset are taken as the training dataset.

3. The fine-grained image classification method based on visual feature constraints according to claim 1 is characterized in that: In the second step, the images are scaled to a uniform size of 224×224, and then data augmentation methods such as random cropping, random flipping, and random Gaussian blurring are used to increase the number of training images.

4. The fine-grained image classification method based on visual feature constraints according to claim 1 is characterized in that: The input of the ViT-B / 16 model in step 3 is to cut the image into 16×16 blocks, and the output is a 768-dimensional visual feature vector of the image. Finally, a multi-layer perceptron is used to obtain the category score of the image.

5. The fine-grained image classification method based on visual feature constraints according to claim 1 is characterized in that: In step 4, the ViT-B / 16 model has 12 layers. The intermediate features output by the last 4, 3, and 2 layers of the pre-trained ViT-B / 16 model are used as the constraint features of image feature constraint 1. The calculation formula of image feature constraint 1 is as follows: <h2 style=";text-align:left;direction:ltr">L1 = f1 - f2<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> , Where f1 is the intermediate feature extracted by the pre-trained model in step 4, and f2 is the corresponding intermediate feature of the model in step 3.

6. The fine-grained image classification method based on visual feature constraints according to claim 1 is characterized in that: In step 5, the text feature dimension extracted by the transformer text feature encoder is 768-dimensional; this text feature is used to perform image feature constraint 2 on the image visual features extracted by the model in step 3; The calculation formula for image feature constraint 2 is as follows: L2=‖f t -favorite i ‖ 2 , where f t is the text feature extracted by the text transformer feature encoder in step 5, f i It is the visual feature of the image extracted by the model in step 3.

7. The fine-grained image classification method based on visual feature constraints according to claim 1 is characterized in that: The pre-trained visual feature encoder used in step 6 obtains an activation map of the training image; the training image is masked using the activation map; specifically, the pixels at the relative positions of the training image with activation map values ​​greater than 0.5 are retained, and the pixels at the relative positions of the training image with activation map values ​​less than 0.5 are set to zero.

8. The fine-grained image classification method based on visual feature constraints according to claim 1 is characterized in that: The common visual features obtained in steps 3 and 7 and the visual features of the mask image are combined in step 8. The calculation formula of the combined features is as follows: f=αf i +(1-a)f m , where f i is the visual feature of the last layer of the image extracted by the model in step 3, f m It is the visual feature of the mask image extracted in step 7, and α is a parameter that adjusts the importance of the two features.

9. The fine-grained image classification method based on visual feature constraints according to claim 1, characterized in that: The rotation angles in step ten are 0°, 90°, 180° and 270° respectively.

Images