A small sample fine-grained image classification method based on prior convolution

Abstract

The application discloses a small sample fine-grained image classification method based on prior convolution and relates to the technical field of image classification. The application firstly constructs an image classification model with multiple prior classifiers, wherein the number of the prior classifiers is equal to the number of image categories; then, pre-training weights are loaded, the model is trained, the best weights of the model are saved, and the training set samples are inferred to obtain prior knowledge; then, a prior convolution branch is added to the prior classifier of the image classification model, and the weights and the prior knowledge are loaded; finally, the model is trained and used for image classification. The application effectively reduces the feature redundancy in the classification model and fully utilizes the feature classification knowledge in the training samples by adding the prior classifier and the innovative training strategy, thereby improving the accuracy and practicability of the small sample fine-grained image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification, and particularly relates to a small sample fine-grained image classification method based on prior convolution. BACKGROUND

[0002] Small sample fine-grained image classification is a challenging task in the field of computer vision, which aims to identify and distinguish subcategories with similar appearance features, such as different types of aircraft, using limited samples. The difficulty of this task lies in the small amount of sample data, high similarity between classes, and large differences within classes, which makes it difficult for traditional image classification methods to effectively distinguish.

[0003] With the development of deep learning technology, convolutional neural networks (CNN) have become the mainstream method for fine-grained image classification. For example, the Bilinear CNN model performs outer product on the outputs of two CNN feature extraction networks to obtain a bilinear vector, thereby achieving fine-grained image classification. In addition, there are methods that propose a graph convolutional neural network-based method that uses label information of graph nodes to implicitly model intra-class similarity and inter-class difference. Although these methods have made some progress in fine-grained image classification, under small sample data conditions, how to reduce network feature redundancy, fully utilize training samples, and enhance network generalization ability remains an open challenge. SUMMARY

[0004] To overcome the shortcomings of the prior art, the present application provides a small sample fine-grained image classification method based on prior convolution. Through innovative network structure and training strategy, the present application effectively reduces feature redundancy in the classification model and fully utilizes classification knowledge in the training samples, improving the accuracy of small sample fine-grained image classification.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] A small sample fine-grained image classification method based on prior convolution, comprising the following steps:

[0007] S1: Establish a fine-grained small sample image classification dataset, which contains images of multiple categories;

[0008] S2: Construct an initial image classification model with multiple prior classifiers; the model uses the backbone and feature fusion layer of the YOLOv8 classification model to extract image features, and connects the prior classifiers equal to the number of image categories after the feature fusion layer. The image features are grouped according to the number of image categories, and each group is sent to the corresponding prior classifier;

[0009] S3: load the pre-training weight, train the initial image classification model using the dataset, save the best weight, and then infer the training samples in the dataset to obtain prior knowledge;

[0010] S4: add a prior convolution branch to the prior classifier of the initial image classification model, load the best weight obtained in step S3 and the prior knowledge, and modify the initial image classification model into an improved image classification model;

[0011] S5: train the improved image classification model;

[0012] S6: classify the image to be classified using the trained improved image classification model.

[0013] Further, the specific way of step S3 is:

[0014] S301: load the official pre-training weight of the initial image classification model, freeze the backbone part for training, then unfreeze the backbone part for training, and save the weight with the highest model accuracy;

[0015] S302: load the weight with the highest model accuracy, infer all training samples in the dataset, when inferring images of the same class, extract the image features after the backbone and the feature fusion layer for each image, cut the features with indexes belonging to the category group, add these cut features to obtain the prior knowledge features.

[0016] Further, the specific way of step S4 is:

[0017] S401: change the structure of the prior classifier, add a new prior convolution branch after the feature fusion layer, and connect the original convolution pooling result and the result of the new prior convolution branch to input into a new fully connected layer to calculate the score of a group of images input into the prior classifier;

[0018] S402: load the model with the highest model accuracy obtained in step S301, assign the parameters of the feature fusion layer and the network layers before it, load the prior knowledge features, and assign each class of prior knowledge features as a convolution kernel parameter to the corresponding prior convolution;

[0019] The output of the improved image classification model is the image category corresponding to the maximum score group, i.e. the predicted classification value of the image.

[0020] Further, the specific way of step S5 is: freeze all parameters of the backbone part, the feature fusion layer, the convolution pooling branch and the prior convolution branch in the prior classifier, train the model, save the best weight, and obtain the trained improved image classification model.

[0021] Compared with the prior art, the present application has the following beneficial effects:

[0022] 1、The prior classifier of the present application enables the model to mine and refine specific features for each category, each prior classifier can focus on feature data related to the category, reduce the mutual interference between different category features, and realize more accurate classification.

[0023] 2、The present application can significantly enhance the recognition ability of the model for key features by embedding prior knowledge, the prior convolution branch parameters of each prior classifier in the model are set as the prior knowledge of the category, which can more accurately capture the features associated with the category, thereby improving the accuracy of recognition. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a schematic diagram of the network structure of the classification model in the embodiment of the present application;

[0025] Figure 2 is a schematic diagram of the structure of the first-stage prior classifier in the embodiment of the present application;

[0026] Figure 3 is a schematic diagram of the structure of the second-stage prior classifier in the embodiment of the present application. DETAILED DESCRIPTION

[0027] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, it should be understood that these examples are only used to illustrate the present application and not used to limit the scope of the present application, after reading the present application, the modifications of various equivalent forms of the present application by those skilled in the art all fall within the scope defined by the appended claims of the present application.

[0028] A small sample fine-grained image classification method based on prior convolution, the specific process is:

[0029] S1: Establish a fine-grained small sample image classification data set, in this example, Mar20 is used as the basic data set, there are 20 categories of aircraft, and 30 slice images are randomly selected for each category as training samples, and 50 slice images are randomly selected as test samples;

[0030] S2: Construct an image classification model with multiple prior classifiers; wherein the number of prior classifiers is equal to the number of image categories, the classification model uses YOLOv8s-cls as the basic image classification model, and uses the backbone and feature fusion layer to extract image features, the specific process is:

[0031] The input image is uniformly scaled to 224x224 and input into the backbone, and the extracted feature F b ∈R 512 ×7×7As input, the input feature fusion layer is used for further feature enhancement and extraction to obtain feature F c ∈R 1280×7×7 , the formula is as follows:

[0032] F c =Conv1(F b )......(1)

[0033] Wherein, Conv1 represents a convolution with a kernel size of 1x1 and a number of 1280.

[0034] YOLOv8s-cls is an open-source efficient target classification model, which converts the input image into a feature map through a series of convolution layers and other operations, and then judges the target category according to these features. For an input image with a size of 224x224, the image is reduced by 5 times to 3x3 through 5 convolution modules, and different numbers of c2f modules are connected after each convolution module to capture multiple details of the target and fuse features at different levels through different convolution operation combinations. Finally, the classify classification head is connected after the last c2f module of the backbone to realize the classification of the image.

[0035] In the image classification model of the embodiment, 20 prior classifiers are connected after the feature fusion layer, as shown in Figure 1 , the prior classifier structure is as shown in Figure 2 . F c is grouped into F c =[f1,f2,...,f 20 ], wherein f i ∈R 64×7×7 , i=1,2,...,20, each group is sent into the corresponding prior classifier for calculation, and the maximum value group of scores in all groups is the image category y c , and the specific process is as follows:

[0036] f i is input into the i-th classifier, and first 3x3 convolution is used to obtain feature , and then pooling is used to obtain feature Then, the full connection layer is used to obtain the score The group with the maximum score in y s is the image category Y c , and the specific formula is as follows:

[0037]

[0038] Y c =Max index (y s )......(3)

[0039] wherein, Conv2 represents a convolution with a kernel size of 3x3, a channel number of 64, and a quantity of 32, Pooling represents pooling, Linear represents a fully connected layer with an input of 32 and an output of 1, Max index represents obtaining the maximum value index, and the operation of the model establishment in the training phase one is completed.

[0040] S3: loading the pre-training weight, training the model and saving the best weight of the model, inferring the training set samples to obtain the prior knowledge. The specific way is:

[0041] S301: loading the YOLOv8s-cls official pre-training weight of the image classification model established in S2, freezing the backbone part for training for 30 rounds, then unfreezing the backbone for training for 70 rounds, and saving the weight with the highest model precision;

[0042] S302: loading the weight with the highest model precision saved in S301, inferring all training samples, and when inferring the same type of image, the result F c ∈R 1280×7×7 after the backbone and the feature fusion layer of each image is obtained, the features of the group with the index belonging to the category are intercepted as F c_i ∈R 64×7×7 , and the features obtained from all images of the category are added, and the final prior knowledge feature of the 20-category training sample image is F pk ∈R 20×64×7×7 .

[0043] S4: adding a prior convolution branch to the prior classifier of the image classification model, and loading the weight and prior knowledge to improve the image classification model constructed in S2. The specific way is:

[0044] S401: improving the structure of the image classification model, the number of the 20 prior classifiers after the feature fusion layer remains unchanged, and the prior classifier structure is changed as shown in Figure 3 , a new prior convolution branch is added after the feature fusion layer, the prior convolution branch is composed of a convolution with a channel number of 64, a kernel size of 7x7, and a quantity of 1, and the original convolution pooling result and the new branch result are connected, input into a new fully connected layer, and the maximum value group of the score is the image category Y c , and the specific process is as follows:

[0045] f i input into the i-th group of classifiers, first obtains the feature through 3x3 convolution, and then obtains the feature At the same time, f i input into the prior convolution branch in the structure of the second training phase, and obtain the similar feature score The two branch results are combined to obtain a feature Scores are obtained through the fully connected layer y s2 The maximum group is the image category Y c , and the specific formula is as follows:

[0046]

[0047] Y c = Max index (y s2 )......(8)

[0048] wherein Conv2 represents a convolution with a kernel size of 3x3, a channel number of 64, and a quantity of 32, Pooling represents pooling, Cat represents concatenating two groups of features in the channel direction, Conv3 represents a convolution with a kernel size of 7x7, a channel number of 64, and a quantity of 1, Linear represents a fully connected layer with an input of 33 and an output of 1, Max index represents obtaining the maximum value index;

[0049] S402: Load the model precision highest weight saved in S301, and assign it to the parameters of the feature fusion layer and the previous network layer, and load the prior knowledge feature F obtained in S302 pk ∈R 20×64×7×7 , a total of 20 64x7x7 dimensional tensors, which are assigned as convolution kernel parameters to 20 groups of prior convolution;

[0050] S5: Train the model, specifically as follows:

[0051] Freeze all parameters in the backbone part, the feature fusion layer, the convolution pooling branch and the prior convolution branch in the prior classifier, train the model, and save the best weight;

[0052] S6: Use the trained improved image classification model to classify the images to be classified, and in this example, the test sample of step S1 is used for inference test, specifically as follows:

[0053] Load the best weight in S5, fix the model parameters, input the test sample image, obtain the predicted classification value, and calculate the precision of the predicted classification value according to the real classification value.

[0054] In this example, the performance of each classification model is compared on the data set established in S1, wherein the precision of the YOLOv8s-cls basic image classification model is 71.4%, the precision of the classification model constructed in S2 is 72.7%, and the precision of the improved classification model in S4 is 73.5%. It can be seen that the image classification model obtained by the method has the highest classification performance.

[0055] In summary, the present application aims at the problems of small sample size, high similarity between classes and large intra-class difference in small sample fine-grained image classification task. By adding a prior classifier and an innovative training strategy, the present application effectively reduces the feature redundancy in the classification model, fully utilizes the feature classification knowledge in the training samples, and improves the accuracy and practicability of small sample fine-grained image classification.

Claims

1. A small sample fine-grained image classification method based on prior convolution, characterized in that, The method comprises the following steps: S1: Establish a fine-grained small sample image classification data set, which contains images of multiple categories; S2: Construct an initial image classification model with multiple prior classifiers; the model uses the backbone and feature fusion layer of the YOLOv8 classification model to extract image features, and connects the prior classifiers equal in number to the image categories after the feature fusion layer; the image features are grouped according to the number of image categories, and each group is sent into the corresponding prior classifier; S3: Load the pre-training weight, train the initial image classification model using the data set, save the best weight, and then infer the training samples in the data set to obtain prior knowledge; the specific method is: S301: Load the official pre-training weight for the initial image classification model, freeze the backbone part for training, then unfreeze the backbone part for training, and save the weight with the highest model accuracy; S302: Load the weight with the highest model accuracy, and infer all training samples in the data set; when inferring images of the same category, the image features extracted after the backbone and the feature fusion layer are intercepted, the features with indexes belonging to the category group are added, and the prior knowledge features are obtained; S4: Add a prior convolution branch to the prior classifier of the initial image classification model, load the best weight and prior knowledge obtained in step S3, and modify the initial image classification model into an improved image classification model; the specific method is: S401: Change the structure of the prior classifier, add a new prior convolution branch after the feature fusion layer, and connect the original convolution pooling result and the result of the new prior convolution branch to the new fully connected layer to calculate the score of a group of images input into the prior classifier; S402: Load the model weight with the highest model accuracy obtained in step S301, assign the parameters of the feature fusion layer and the network layers before it, load the prior knowledge features, and assign each category of prior knowledge features as a convolution kernel parameter to the corresponding prior convolution; The output of the improved image classification model is the image category corresponding to the maximum score group, that is, the predicted classification value of the image; S5: Train the improved image classification model; S6: Classify the image to be classified using the trained improved image classification model.

2. The small sample fine-grained image classification method based on prior convolution according to claim 1, characterized in that, The specific method of step S5 is to freeze all parameters of the backbone part, the feature fusion layer, the convolution pooling branch and the prior convolution branch in the prior classifier, train the model, save the best weight, and obtain the trained improved image classification model.

Images