Small sample fine-grained image classification method based on prior convolution

Abstract

The invention discloses a small sample fine-grained image classification method based on prior convolution, and relates to the technical field of image classification. The method comprises the following steps: firstly, constructing an image classification model with a plurality of prior classifiers, wherein the number of the prior classifiers is equal to the number of image categories; then loading a pre-training weight, training a model and storing the optimal weight of the model, and reasoning a training set sample to obtain priori knowledge; adding a priori convolution branch into a priori classifier of the image classification model, and loading a weight and priori knowledge; and finally training the model for image classification. Aiming at the problems of small sample size, high inter-class similarity and large intra-class difference in a small-sample fine-grained image classification task, by adding a prior classifier and an innovative training strategy, feature redundancy in a classification model is effectively reduced, feature classification knowledge in a training sample is fully utilized, and the classification efficiency of the small-sample fine-grained image is improved. And the accuracy and practicability of small sample fine-grained image classification are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a small sample fine-grained image classification method based on prior convolution. Background Art

[0002] Small sample fine-grained image classification is a challenging task in the field of computer vision. Its purpose is to use limited samples to identify and distinguish subcategories with similar appearance features, such as different types of aircraft. The difficulty of this task lies in the small number of sample data, high similarity between categories, and large differences within categories, which makes it difficult for traditional image classification methods to effectively distinguish.

[0003] With the development of deep learning technology, convolutional neural networks (CNN) have become the mainstream method for fine-grained image classification. For example, the Bilinear CNN model obtains a bilinear vector by performing an outer product of the outputs of two CNN feature extraction networks, thereby achieving fine-grained image classification. In addition, there are methods based on graph convolutional neural networks that use the label information of graph nodes to implicitly model intra-class similarities and inter-class differences. Although these methods have made some progress in fine-grained image classification, it is still an open challenge to reduce network feature redundancy, make full use of training samples, and enhance the generalization ability of the network in the case of small sample data. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a small sample fine-grained image classification method based on prior convolution. The present invention effectively reduces the feature redundancy in the classification model and fully utilizes the classification knowledge in the training samples through innovative network structure and training strategy, thereby improving the accuracy of small sample fine-grained image classification.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A small sample fine-grained image classification method based on prior convolution includes the following steps:

[0007] S1: Establish a fine-grained small sample image classification dataset, which contains images of multiple categories;

[0008] S2: Build an initial image classification model with multiple prior classifiers. This model uses the backbone and feature fusion layer of the YOLOv8 classification model to extract image features. After the feature fusion layer, a priori classifiers equal to the number of image categories are connected. The image features are grouped according to the number of image categories, and each group is sent to the corresponding priori classifier.

[0009] S3: Load the pre-trained weights, use the dataset to train the initial image classification model, save the best weights, and then perform inference on the training samples in the dataset to obtain prior knowledge;

[0010] S4: adding a priori convolution branch to the priori classifier of the initial image classification model, and loading the best weight and priori knowledge obtained in step S3, to modify the initial image classification model into an improved image classification model;

[0011] S5: training the improved image classification model;

[0012] S6: Use the trained improved image classification model to classify the image to be classified.

[0013] Furthermore, the specific method of step S3 is:

[0014] S301: Load the official pre-trained weights for the initial image classification model, first freeze the backbone part for training, then unfreeze the backbone part for training, and save the weights with the highest model accuracy;

[0015] S302: Load the model with the highest accuracy weight and perform inference on all training samples in the data set. When inferring the same type of images, extract the image features of each image after the backbone and feature fusion layer, extract the features whose index belongs to the category group, and add these extracted features to obtain the prior knowledge features.

[0016] Furthermore, the specific method of step S4 is:

[0017] S401: changing the structure of the prior classifier, adding a new prior convolution branch after the feature fusion layer, and connecting the original convolution pooling result with the result of the new prior convolution branch, inputting them into a new fully connected layer, and calculating the scores of a group of images input into the prior classifier;

[0018] S402: Load the weight with the highest model accuracy obtained in step S301, assign it to the parameters of the feature fusion layer and the network layer before it, load the prior knowledge features, and assign each type of prior knowledge features as convolution kernel parameters to the prior convolution of the corresponding category;

[0019] The output of the improved image classification model is the image category corresponding to the maximum score group, that is, the predicted classification value of the image.

[0020] Furthermore, the specific method of step S5 is: freezing all parameters of the backbone part, the feature fusion layer, the convolution pooling branch and the prior convolution branch in the prior classifier, training the model, and saving the optimal weights to obtain a trained improved image classification model.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The prior classifier of the present invention enables the model to mine and refine specific features for each category. Each prior classifier can focus on the feature data related to the category, reduce the mutual interference between features of different categories, and achieve more accurate classification.

[0023] 2. The present invention can significantly enhance the model's ability to recognize key features by embedding prior knowledge. The prior convolution branch parameters of each prior classifier in the model are set to this type of prior knowledge, which can more accurately capture the features associated with the category, thereby improving the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a network structure of a classification model in an embodiment of the present invention;

[0025] Figure 2 Schematic diagram of the structure of the first stage a priori classifier in an embodiment of the present invention;

[0026] Figure 3 Schematic diagram of the structure of the second stage a priori classifier in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0028] A small sample fine-grained image classification method based on prior convolution, the specific process is:

[0029] S1: Establish a fine-grained small sample image classification dataset. In this example, Mar20 is used as the basic dataset, with a total of 20 types of aircraft. For each type, 30 slice images are randomly selected as training samples, and 50 slice images are randomly selected as test samples.

[0030] S2: Build an image classification model with multiple prior classifiers. The number of prior classifiers is equal to the number of image categories. The classification model uses YOLOv8s-cls as the basic image classification model, and uses its backbone and feature fusion layer to extract image features. The specific process is as follows:

[0031] The input image is uniformly scaled to 224×224 and input into the backbone. The extracted features F b ∈R 512 ×7×7As input, the feature fusion layer is used to further enhance and extract features to obtain feature F c ∈R 1280×7×7 , the formula is as follows:

[0032] F c =Conv1(F b )......(1)

[0033] Among them, Conv1 represents a convolution with a convolution kernel size of 1×1 and a number of 1280.

[0034] YOLOv8s-cls is an open-source, efficient target classification model. It converts the input image into a feature map through a series of convolutional layers and other operations, and then determines the category of the target based on these features. For an input image of size 224×224, it reduces the image by 5 times to 3×3 through 5 convolutional modules, and connects different numbers of c2f modules after each convolutional module. It captures various details of the target and fuses features at different levels through different combinations of convolution operations. Finally, it connects the classify classification head after the last c2f module of the backbone to classify the image.

[0035] In the image classification model of this embodiment, 20 prior classifiers are connected after the feature fusion layer, such as Figure 1 As shown, the prior classifier structure is as follows Figure 2 As shown. c Grouped as F c =[f1,f2,...,f 20 ], where f i ∈R 64×7×7 , i = 1, 2, ..., 20, each group is sent to the corresponding prior classifier for calculation, and the group with the maximum score among all groups is the image category y c , the specific process is:

[0036] f i Input the i-th group of classifiers, first obtain the features through 3×3 convolution Then obtain features through pooling Then pass the fully connected layer to get the score y s The group with the largest score is image category Y c , the specific formula is as follows:

[0037]

[0038] Y c =Max index (y s )......(3)

[0039] Among them, Conv2 represents a convolution with a kernel size of 3×3, 64 channels, and 32 counts, Pooling represents pooling, Linear represents a fully connected layer with an input of 32 and an output of 1, and Max index It indicates that the maximum value index is obtained, and the operation of building a model in the training phase is completed.

[0040] S3: Load pre-trained weights, train the model and save the best model weights, perform inference on the training set samples, and obtain prior knowledge. The specific method is:

[0041] S301: Load the YOLOv8s-cls official pre-trained weights for the image classification model established in S2, freeze the backbone part for 30 rounds of training, then unfreeze the backbone for 70 rounds of training, and save the weights with the highest model accuracy;

[0042] S302: Load the model with the highest accuracy weight saved in S301, infer all training samples, and when inferring the same type of images, calculate the result F of each image after the backbone and feature fusion layer. c ∈R 1280×7×7 , intercept the features of the group whose index belongs to this category as F c_i ∈R 64×7×7 , add up the features of all images in this category, and finally the prior knowledge features of the 20 categories of training sample images are obtained as F pk ∈R 20×64×7×7 .

[0043] S4: Add a priori convolution branch to the priori classifier of the image classification model, and load weights and priori knowledge to improve the image classification model constructed in S2. The specific method is:

[0044] S401: Improve the structure of the image classification model. The number of 20 prior classifiers after the feature fusion layer remains unchanged. Change the structure of the prior classifier as follows: Figure 3 As shown in the figure, a new prior convolution branch is added after the feature fusion layer. The prior convolution branch consists of a convolution with 64 channels, a convolution kernel size of 7×7, and a number of 1. The original convolution pooling result is connected with the new branch result and input into the new fully connected layer. The maximum score group is the image category Y c , the specific process is as follows:

[0045] f i Input the i-th group of classifiers, first obtain the features through 3×3 convolution Then obtain features through pooling At the same time, f i Input the prior convolution branch in the second structure of the training stage to obtain the similarity feature score Combine the two branch results to obtain the feature Get the score after the fully connected layer y s2 The largest group is image category Y c , the specific formula is as follows:

[0046]

[0047] Y c =Max index (y s2 )......(8)

[0048] Among them, Conv2 represents a convolution with a kernel size of 3×3, 64 channels, and 32 counts. Pooling represents pooling. Cat represents two sets of input features spliced ​​in the channel direction. Conv3 represents a convolution with a kernel size of 7×7, 64 channels, and 1 count. Linear represents a fully connected layer with 33 inputs and 1 output. Max index Indicates obtaining the maximum value index;

[0049] S402: Load the highest model accuracy weight saved in S301, assign it to the network layer parameters including the feature fusion layer and the previous one, and load the prior knowledge feature F obtained in S302 pk ∈R 20×64×7×7 , a total of 20 64×7×7 dimensional tensors are assigned as convolution kernel parameters to the prior convolutions of 20 groups of categories;

[0050] S5: Train the model. The specific method is as follows:

[0051] Freeze all parameters of the backbone part, feature fusion layer, convolutional pooling branch in the prior classifier, and prior convolutional branch, train the model, and save the best weights;

[0052] S6: Use the trained improved image classification model to classify the image to be classified. In this example, the test sample in step S1 is used for reasoning test. The specific method is as follows:

[0053] Load the best weights in S5, fix the model parameters, input the test sample image, obtain the predicted classification value, and calculate the accuracy of the predicted classification value based on the true classification value.

[0054] In this example, the performance of each classification model is compared on the data set established by S1. The accuracy of the YOLOv8s-cls basic image classification model is 71.4%, the accuracy of the classification model constructed by S2 is 72.7%, and the accuracy of the improved classification model of S4 is 73.5%. It can be seen that the image classification model finally obtained by this method has the highest classification performance.

[0055] In summary, the present invention addresses the problems of small sample size, high similarity between categories, and large intra-category differences in small-sample fine-grained image classification tasks. By adding prior classifiers and innovative training strategies, it effectively reduces the feature redundancy in the classification model and makes full use of the feature classification knowledge in the training samples, thereby improving the accuracy and practicality of small-sample fine-grained image classification.

Claims

1. A small sample fine-grained image classification method based on prior convolution, characterized in that: The following steps are involved: S1: Establish a fine-grained small sample image classification dataset, which contains images of multiple categories; S2: Build an initial image classification model with multiple prior classifiers. This model uses the backbone and feature fusion layer of the YOLOv8 classification model to extract image features. After the feature fusion layer, a priori classifiers equal to the number of image categories are connected. The image features are grouped according to the number of image categories, and each group is sent to the corresponding priori classifier. S3: Load the pre-trained weights, use the dataset to train the initial image classification model, save the best weights, and then perform inference on the training samples in the dataset to obtain prior knowledge; S4: adding a priori convolution branch to the priori classifier of the initial image classification model, and loading the best weight and priori knowledge obtained in step S3, to modify the initial image classification model into an improved image classification model; S5: training the improved image classification model; S6: Use the trained improved image classification model to classify the image to be classified.

2. According to the method of small sample fine-grained image classification based on prior convolution according to claim 1, it is characterized in that: The specific method of step S3 is: S301: Load the official pre-trained weights for the initial image classification model, first freeze the backbone part for training, then unfreeze the backbone part for training, and save the weights with the highest model accuracy; S302: Load the model with the highest accuracy weight and perform inference on all training samples in the data set. When inferring the same type of images, extract the image features of each image after the backbone and feature fusion layer, extract the features whose index belongs to the category group, and add these extracted features to obtain the prior knowledge features.

3. According to claim 2, a small sample fine-grained image classification method based on prior convolution is characterized in that: The specific method of step S4 is: S401: changing the structure of the prior classifier, adding a new prior convolution branch after the feature fusion layer, and connecting the original convolution pooling result with the result of the new prior convolution branch, inputting them into a new fully connected layer, and calculating the scores of a group of images input into the prior classifier; S402: Load the weight with the highest model accuracy obtained in step S301, assign it to the parameters of the feature fusion layer and the network layer before it, load the prior knowledge features, and assign each type of prior knowledge features as convolution kernel parameters to the prior convolution of the corresponding category; The output of the improved image classification model is the image category corresponding to the maximum score group, that is, the predicted classification value of the image.

4. The small sample fine-grained image classification method based on prior convolution according to claim 3 is characterized in that: The specific method of step S5 is: freezing all parameters of the backbone part, the feature fusion layer, the convolution pooling branch and the prior convolution branch in the prior classifier, training the model, and saving the optimal weights to obtain a trained improved image classification model.

Images