A small-sample fine-grained image classification method based on bimetric networks

Abstract

A kind of small sample fine-grained image classification method based on double metric network, by constructing small sample data set, constructing double metric network, training double metric network, save model, verify double metric network, test double metric network step composition, the method uses space-channel attention module to make network focus on the key area with distinguishing force in image, using global feature metric network and local feature metric network, respectively determine global similarity and local similarity based on local descriptor for enhanced feature, and the final similarity is obtained by combining two kinds of similarity to classify.In the training network step, triad loss function and cross-entropy loss function are used for training, solve the problem that existing method is prone to overfitting when data volume is insufficient.The present application has the advantages of small sample amount, fast classification speed, high classification accuracy and the like, and can be used for small sample fine-grained image classification.

Description

Technical Field

[0001] This invention relates to the field of computer vision recognition, and more specifically to image classification methods. Background Technology

[0002] The rapid development of artificial intelligence has led to significant progress in image classification technology based on deep learning in computer vision. However, deep learning models suffer from drawbacks such as complex network structures, a large number of parameters, and high computational costs, making their performance heavily reliant on large amounts of labeled data. In practical applications, researchers often struggle to obtain sufficient labeled data for training models in many fields. When data is insufficient, models frequently overfit, resulting in decreased learning and generalization abilities.

[0003] Traditional image classification categorizes objects into different types, such as dogs and cats, which are relatively easy to classify due to their obvious differences. Fine-grained image classification, however, requires more detailed classification within the same category, such as distinguishing bird species, car models, and dog breeds. Currently, most fine-grained image classification methods require training on large-scale labeled data. If the dataset is too small, it can easily lead to overfitting and a significant drop in performance.

[0004] In the field of image classification technology, in the face of the above-mentioned technical problems, it is necessary to change the way of training models with a large amount of data. At present, one of the urgent technical problems to be solved is to provide a fine-grained image classification method that uses a small number of samples, has a fast classification speed, and a high classification accuracy. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a small sample fine-grained image classification method based on a dual-metric network that uses a small sample size, has a fast classification speed, and a high classification accuracy.

[0006] The technical solution adopted to solve the above technical problems involves the following steps:

[0007] (1) Constructing a small sample dataset

[0008] The public datasets CUB-200, StanfordDogs, and StanfordCars, as well as the self-built dataset CIIP-TPID, were used as experimental datasets. The images in the CUB-200, StanfordDogs, and StanfordCars datasets were sized to 84×84 pixels, and the images in the CIIP-TPID dataset were sized to 48×48 pixels. The datasets were divided into training, validation, and test sets in a ratio of 8:1:1. The image categories in the training, validation, and test sets did not overlap, and the images in each category were distinct.

[0009] In the training set, validation set, and test set, W image categories are randomly selected. From each of the W categories, K+X samples are randomly selected, where X is a non-zero positive integer. The K samples are used as the support set S, and the X samples are used as the query set Q. The support set S and the query set Q have the same categories but different samples within each category. One support set S and one query set Q constitute one classification unit. The training set, validation set, and test set are divided into R classification units, where W, K, X, S, Q, and R are finite positive integers.

[0010] (2) Constructing a dual-metric network

[0011] The dual-metric network consists of a feature embedding module and a spatial-channel attention module connected in series. The output of the spatial-channel attention module is connected to the global feature metric network and the local feature metric network, respectively.

[0012] The feature embedding module consists of a convolutional layer 1, 4 to 12 residual blocks, an average pooling layer 1, a batch normalization layer 1, and a fully connected layer 1 connected in series.

[0013] (3) Training the dual-metric network

[0014] 1) Determine the objective function

[0015] The objective function includes the triplet loss function L1, the cross-entropy loss function L2, and the evaluation function ACC; the triplet loss function L1 is determined by the following formula:

[0016]

[0017] Where a represents the query sample, p represents the sample belonging to the query set category, and n represents the sample that does not belong to the query set category. This represents different samples in the feature embedding module, where N is the total number of samples and is a finite positive integer.

[0018] The cross-entropy loss function L2 is determined by the following formula:

[0019]

[0020] Among them, y i It is the true label of the sample, p i It is the probability that the sample belongs to the i-th class as predicted by the network model.

[0021] The evaluation function ACC is determined by the following formula:

[0022]

[0023] Where TP represents the number of correctly predicted positive examples, FN represents the number of incorrectly predicted positive examples, TN represents the number of correctly predicted negative examples, FP represents the number of incorrectly predicted negative examples, the sum of TP and FP, TN, and FN is the total number of samples, the sum of TP and TN is the number of correctly predicted samples, the sum of FN and FP is the number of incorrectly predicted samples, TP, FP, TN, and FN ∈ M, M is a finite positive integer, and TP, FP, TN, and FN are not all 0 at the same time.

[0024] 2) Training a dual-metric network

[0025] The support set S and query set Q from the training set are input into the dual-metric network for training. The Adam optimizer is used during training, with the learning rate initialized to 0.001. 150 training epochs are performed, with 100 parameter learning iterations in each epoch. The learning rate is halved every 30 training epochs. The overfitting phenomenon during the training process of the dual-metric network is monitored using an early termination method.

[0026] (4) Save the model

[0027] During the training of the dual-metric network, the weights are continuously updated and the weight file is saved.

[0028] (5) Validate the dual-metric network

[0029] The support set S and query set Q from the validation set are input into the trained dual-metric network for validation.

[0030] (6) Testing the dual-metric network

[0031] The support set S and query set Q from the test set are input into the trained dual-metric network for testing. The saved weight file is loaded to obtain the image classification.

[0032] In step (2) of the present invention, the residual block of the feature embedding module is composed of convolutional layer 2, activation function layer 1, and convolutional layer 3 connected in series.

[0033] In step (2) of the present invention, the construction of the dual-metric network is made by connecting the position attention module and the channel attention module in parallel and then connecting them in series with the convolutional layer 4.

[0034] In step (2) of the present invention, the global feature measurement network is constructed by sequentially connecting an embedded module and a correlation module, a fully connected layer 2, a fully connected layer 3, and a Sigmoid function layer.

[0035] In step (2) of the present invention, the local feature measurement network is constructed by convolutional embedding module and covariance measurement module connected in series.

[0036] In step (2) of the present invention, the embedded module of the global feature measurement network is composed of four convolutional layers connected in series; the correlation module of the global feature measurement network is composed of convolutional layer 6 connected in series with max pooling layer 1, convolutional layer 7, and max pooling layer 2.

[0037] In step (2) of the present invention, the construction of the dual-metric network is carried out by the convolutional embedding module of the local feature metric network, which is composed of convolutional layer 8, normalization layer 1, activation function layer 2, max pooling layer 3, convolutional layer 9, normalization layer 2, activation function layer 3, max pooling layer 4, convolutional layer 10, normalization layer 3, activation function layer 4, convolutional layer 11, normalization layer 4, and activation function layer 5 connected in series.

[0038] In step (1) of the present invention, in constructing a small sample dataset, W image categories are randomly selected from the training set, validation set, and test set, where W is 5 or 10. From the W categories, K+X samples are randomly selected, where K is 1 or 5 and X is a positive integer not equal to 0. The K samples are used as the support set S and the X samples are used as the query set Q. The support set S and the query set Q have the same category but different in-class samples. A support set S and a query set Q constitute a classification unit. The training set, validation set, and test set are divided into R classification units, where X, S, Q, and R are finite positive integers.

[0039] Because this invention employs a global metric network, it avoids the difficulty of manually selecting metric methods, enabling the network to adapt to different images for classification. In the local feature metric network, a covariance representation and covariance metric based on distribution consistency are introduced to measure the similarity between local descriptors of support set samples and query set samples. The final similarity result is derived from global and local similarities. This metric method considers not only the semantic information of global features and the detailed information of local features, but also combines first-order mean information and second-order covariance information for measurement. For the characteristics of fine-grained images, an improved triplet loss function optimization model is introduced, comprehensively considering the support set feature distribution, enabling the network to learn better decision boundaries, improving the network's classification ability, and generating richer, more discriminative features. This invention has the advantages of using a small sample size, fast classification speed, and high classification accuracy, and can be used for small-sample, fine-grained image classification. Attached Figure Description

[0040] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0041] Figure 2 This is a schematic diagram of the structure of a dual-metric network.

[0042] Figure 3 yes Figure 2A schematic diagram of the feature embedding module.

[0043] Figure 4 yes Figure 3 A schematic diagram of the structure of the residual block.

[0044] Figure 5 yes Figure 2 A schematic diagram of the structure of the mid-space-channel attention module.

[0045] Figure 6 yes Figure 2 A schematic diagram of the structure of the global feature measurement network.

[0046] Figure 7 yes Figure 6 A schematic diagram of the embedded module.

[0047] Figure 8 yes Figure 6 A schematic diagram of the structure of the correlation module.

[0048] Figure 9 yes Figure 2 A schematic diagram of the convolutional embedding module of the local feature measurement network. Detailed Implementation

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments described below.

[0050] Example 1

[0051] exist Figure 1 In this embodiment, the small-sample fine-grained image classification method based on dual-metric networks consists of the following steps.

[0052] (1) Constructing a small sample dataset

[0053] The public datasets CUB-200, StanfordDogs, and StanfordCars, as well as the self-built dataset CIIP-TPID, were used as experimental datasets. The images in the CUB-200, StanfordDogs, and StanfordCars datasets were sized to 84×84 pixels, and the images in the CIIP-TPID dataset were sized to 48×48 pixels. The datasets were divided into training, validation, and test sets in a ratio of 8:1:1. The image categories in the training, validation, and test sets did not overlap, and the images in each category were distinct.

[0054] In the training set, validation set, and test set, N image categories are randomly selected, and W takes the value of 5 or 10. In this embodiment, W takes the value of 5. From the W categories, K+X samples are randomly selected, and K takes the value of 1 or 5. In this embodiment, K takes the value of 1. X is a positive integer that is not 0. The K samples are used as the support set S, and the X samples are used as the query set Q. The support set S and the query set Q have the same category, but the samples within the same category are different. A support set S and a query set Q constitute a classification unit. The training set, validation set, and test set are divided into R classification units, where X, S, Q, and R are finite positive integers.

[0055] (2) Constructing a dual-metric network

[0056] Figure 2 A schematic diagram of the dual-metric network structure is given. Figure 2 In this embodiment, the dual-metric network consists of a feature embedding module and a spatial-channel attention module connected in series. The output of the spatial-channel attention module is connected to the global feature metric network and the local feature metric network, respectively.

[0057] Figure 3 Given Figure 2 A schematic diagram of the feature embedding module is shown in the figure. Figure 3 In this embodiment, the feature embedding module is composed of a convolutional layer 1, 4 to 12 residual blocks, an average pooling layer 1, a batch normalization layer 1, and a fully connected layer 1 connected in series. The feature embedding module in this embodiment has 8 residual blocks, and the convolutional kernel size of the convolutional layer 1 is 7×7 with a stride of 2.

[0058] Figure 4 Given Figure 3 A schematic diagram of the structure of the residual block. Figure 4 In this embodiment, the residual block of the feature embedding module is composed of convolutional layer 2, activation function layer 1, and convolutional layer 3 connected in series. In this embodiment, the convolutional kernel size of convolutional layer 2 is 3×3 with a stride of 2, and the convolutional kernel size of convolutional layer 3 is 3×3 with a stride of 1.

[0059] Figure 5 Given Figure 2 A schematic diagram of the mid-space-channel attention module. Figure 5 In this embodiment, the spatial-channel attention module is composed of a position attention module and a channel attention module connected in parallel, and then connected in series with the convolutional layer 4. The convolutional kernel size of the convolutional layer 4 in this embodiment is 3×3 and the stride is 1.

[0060] The channel attention module M in this embodiment j Build as follows:

[0061]

[0062]

[0063] Among them, X ji Let F represent the similarity between the i-th channel and the j-th channel. i For the feature map in the i-th channel, F j Let be the feature maps in the j-th channel, C be the total number of channels, C be a finite positive integer, and β be the network weight, which can be 0 initially.

[0064] Figure 6 Given Figure 2 A schematic diagram of the structure of the global feature measurement network. Figure 6 In this embodiment, the global feature measurement network is composed of an embedded module and a correlation module, a fully connected layer 2, a fully connected layer 3, and a Sigmoid function layer connected in series.

[0065] The local feature measurement network in this embodiment is composed of a convolutional embedding module and a covariance measurement module connected in series.

[0066] The covariance measurement module in this embodiment is constructed using the following formula:

[0067]

[0068]

[0069] Where x represents the set of local descriptors of the query set image, z contains local similarities between a query image and a category, and diag(·) returns the column vector of the main diagonal elements of the matrix. i Let τ be the initial feature map, and τ represent the mean vector of the local descriptors. MK represents the covariance of a class of images, where MK is the total number of feature maps and its value is a positive integer other than 1.

[0070] Figure 7 Given Figure 6 A schematic diagram of the embedded module structure of the global feature measurement network. Figure 7 In this embodiment, the embedded module of the global feature measurement network consists of four convolutional layers connected in series. The kernel size of the four convolutional layers is 3×3 and the stride is 1.

[0071] Figure 8 Given Figure 6 A structural diagram of the correlation module. Figure 8 In this embodiment, the correlation module of the global feature measurement network is composed of convolutional layer 6, max pooling layer 1, convolutional layer 7, and max pooling layer 2 connected in series. The convolutional kernels of convolutional layers 6 and 7 are 3×3 with a stride of 1, while the convolutional kernels of max pooling layers 1 and 2 are 2×2.

[0072] Figure 9 Given Figure 2 A schematic diagram of the convolutional embedding module in the local feature measurement network. Figure 9 In this embodiment, the convolutional embedding module of the local feature measurement network is composed of a convolutional layer 8, a normalization layer 1, an activation function layer 2, a max pooling layer 3, a convolutional layer 9, a normalization layer 2, an activation function layer 3, a max pooling layer 4, a convolutional layer 10, a normalization layer 3, an activation function layer 4, a convolutional layer 11, a normalization layer 4, and an activation function layer 5 connected in series.

[0073] Because this invention employs a global metric network, it avoids the difficulty of manually selecting metric methods, enabling the network to adapt to different images for classification. In the local feature metric network, a covariance representation and covariance metric based on distribution consistency are introduced to assess the similarity between local descriptors of support set samples and query set samples. The final similarity result is derived from global and local similarities. This metric method considers not only the semantic information of global features and the detailed information of local features, but also combines first-order mean information and second-order covariance information for measurement.

[0074] (3) Training the dual-metric network

[0075] 1) Determine the objective function

[0076] The objective function includes the triplet loss function L1, the cross-entropy loss function L2, and the evaluation function ACC; the triplet loss function L1 is determined by the following formula:

[0077]

[0078] Where a represents the query sample, p represents the sample belonging to the query set category, and n represents the sample that does not belong to the query set category. This represents different samples in the feature embedding module, where N is the total number of samples and is a finite positive integer.

[0079] The cross-entropy loss function L2 is determined by the following formula:

[0080]

[0081] Among them, y i It is the true label of the sample, p i It is the probability that the sample belongs to the i-th class as predicted by the network model.

[0082] This invention employs a triplet loss function and a cross-entropy loss function for training, addressing the overfitting problem inherent in existing methods when data is insufficient. By comprehensively considering the support set feature distribution, the network learns better decision boundaries, thereby improving its classification ability.

[0083] The evaluation function ACC is determined by the following formula:

[0084]

[0085] Where TP represents the number of correctly predicted positive examples, FN represents the number of incorrectly predicted positive examples, TN represents the number of correctly predicted negative examples, FP represents the number of incorrectly predicted negative examples, the sum of TP and FP, TN, and FN is the total number of samples, the sum of TP and TN is the number of correctly predicted samples, the sum of FN and FP is the number of incorrectly predicted samples, TP, FP, TN, and FN ∈ M, M is a finite positive integer, and TP, FP, TN, and FN are not all 0 at the same time.

[0086] 2) Training a dual-metric network

[0087] The support set S and query set Q from the training set are input into the dual-metric network for training. The Adam optimizer is used during training, with the learning rate initialized to 0.001. 150 training epochs are performed, with 100 parameter learning iterations in each epoch. The learning rate is halved every 30 training epochs. An early termination method is used to monitor overfitting during the training process of the dual-metric network. Training continues until the dual-metric network converges.

[0088] (4) Save the model

[0089] During the training of the dual-metric network, the weights are continuously updated and the weight file is saved.

[0090] (5) Validate the dual-metric network

[0091] The support set S and query set Q from the validation set are input into the trained dual-metric network for validation.

[0092] (6) Testing the dual-metric network

[0093] The support set S and query set Q from the test set are input into the trained dual-metric network for testing. The saved weight file is loaded to obtain the image classification.

[0094] Complete a small-sample fine-grained image classification method based on a dual-metric network.

[0095] Example 2

[0096] The small-sample fine-grained image classification method based on dual-metric networks in this embodiment consists of the following steps.

[0097] (1) Constructing a small sample dataset

[0098] The public datasets CUB-200, StanfordDogs, and StanfordCars, as well as the self-built dataset CIIP-TPID, were used as experimental datasets. The images in the CUB-200, StanfordDogs, and StanfordCars datasets were sized to 84×84 pixels, and the images in the CIIP-TPID dataset were sized to 48×48 pixels. The datasets were divided into training, validation, and test sets in a ratio of 8:1:1. The image categories in the training, validation, and test sets did not overlap, and the images in each category were distinct.

[0099] In the training set, validation set, and test set, N image categories are randomly selected, and W is either 5 or 10. In this embodiment, W is 10. From each of the W categories, K+X samples are randomly selected, where K is either 1 or 5. In this embodiment, K is 5, and X is a non-zero positive integer. The K samples are used as the support set S, and the X samples are used as the query set Q. The support set S and the query set Q have the same categories but different samples within each category. A support set S and a query set Q constitute a classification unit. The training set, validation set, and test set are divided into R classification units, where X, S, Q, and R are finite positive integers.

[0100] (2) Constructing a dual-metric network

[0101] The dual-metric network consists of a feature embedding module and a spatial-channel attention module connected in series. The output of the spatial-channel attention module is connected to the global feature metric network and the local feature metric network, respectively.

[0102] The feature embedding module of this embodiment consists of a convolutional layer 1, 4 to 12 residual blocks, an average pooling layer 1, a batch normalization layer 1, and a fully connected layer 1 connected in series. The feature embedding module of this embodiment has 4 residual blocks, and the convolutional kernel size of the convolutional layer 1 is 7×7 with a stride of 2.

[0103] The other steps are the same as in Example 1.

[0104] Complete a small-sample fine-grained image classification method based on a dual-metric network.

[0105] Example 3

[0106] The small-sample fine-grained image classification method based on dual-metric networks in this embodiment consists of the following steps.

[0107] (1) Constructing a small sample dataset

[0108] The public datasets CUB-200, StanfordDogs, and StanfordCars, as well as the self-built dataset CIIP-TPID, were used as experimental datasets. The images in the CUB-200, StanfordDogs, and StanfordCars datasets were sized to 84×84 pixels, and the images in the CIIP-TPID dataset were sized to 48×48 pixels. The datasets were divided into training, validation, and test sets in a ratio of 8:1:1. The image categories in the training, validation, and test sets did not overlap, and the images in each category were distinct.

[0109] In the training set, validation set, and test set, N image categories are randomly selected, and W takes the value of 5 or 10. In this embodiment, W takes the value of 5. From the W categories, K+X samples are randomly selected, and K takes the value of 1 or 5. In this embodiment, K takes the value of 5. X is a positive integer that is not 0. The K samples are used as the support set S, and the X samples are used as the query set Q. The support set S and the query set Q have the same category, but the samples within the same category are different. A support set S and a query set Q constitute a classification unit. The training set, validation set, and test set are divided into R classification units, where W, K, X, S, Q, and R are finite positive integers.

[0110] (2) Constructing a dual-metric network

[0111] The dual-metric network consists of a feature embedding module and a spatial-channel attention module connected in series. The output of the spatial-channel attention module is connected to the global feature metric network and the local feature metric network, respectively.

[0112] The feature embedding module of this embodiment consists of a convolutional layer 1, 4 to 12 residual blocks, an average pooling layer 1, a batch normalization layer 1, and a fully connected layer 1 connected in series. The feature embedding module of this embodiment has 12 residual blocks, and the convolutional kernel size of the convolutional layer 1 is 7×7 with a stride of 2.

[0113] The other steps are the same as in Example 1.

[0114] Complete a small-sample fine-grained image classification method based on a dual-metric network.

[0115] Example 4

[0116] The small-sample fine-grained image classification method based on dual-metric networks in this embodiment consists of the following steps.

[0117] (1) Constructing a small sample dataset

[0118] The public datasets CUB-200, StanfordDogs, and StanfordCars, as well as the self-built dataset CIIP-TPID, were used as experimental datasets. The images in the CUB-200, StanfordDogs, and StanfordCars datasets were sized to 84×84 pixels, and the images in the CIIP-TPID dataset were sized to 48×48 pixels. The datasets were divided into training, validation, and test sets in a ratio of 8:1:1. The image categories in the training, validation, and test sets did not overlap, and the images in each category were distinct.

[0119] In the training set, validation set, and test set, N image categories are randomly selected, and W takes the value of 5 or 10. In this embodiment, W takes the value of 10. From the W categories, K+X samples are randomly selected, and K takes the value of 1 or 5. In this embodiment, K takes the value of 1. X is a positive integer that is not 0. The K samples are used as the support set S, and the X samples are used as the query set Q. The support set S and the query set Q have the same category, but the samples within the same category are different. A support set S and a query set Q constitute a classification unit. The training set, validation set, and test set are divided into R classification units, where X, S, Q, and R are finite positive integers.

[0120] The other steps are the same as in Example 1.

[0121] Complete a small-sample fine-grained image classification method based on a dual-metric network.

[0122] To verify the beneficial effects of this invention, the few-shot fine-grained image classification method based on a dual-metric network, as described in Embodiment 1 of this invention, was used to conduct an accuracy comparison experiment with classic few-shot image classification methods and few-shot learning methods for fine-grained image classification using the CUB-200, StanfordDogs, StanfordCars, and CIIP-TPID fine-grained datasets as experimental data. Simultaneously, the MiniImageNet dataset was used as experimental data to compare the classification accuracy with few-shot image classification methods based on meta-learning, metric learning, and fine-tuning, performing experiments with one sample per of five image classes.

[0123] During the implementation, the experimental environment used Ubuntu 16.04 operating system, the programming environment was based on Python 3.6, the graphics card was NVIDIA TITAN NSP GPUs, the CPU information was Intel(R) Xeon(R) CPU E5-2620v4@2.10GHz, the CUDA version was 10.0, and the deep learning framework was PyTorch version 1.0.0-GPU.

[0124] The method of this invention was compared with the following algorithms on the MiniImageNet dataset: MatchingNet, ProtoNet, RelationNet, CovaMNet (a few-shot learning algorithm based on metric learning), DN4 (deep nearest neighbor neural network algorithm), CAN (cross-attention network algorithm), MAML (meta-learning-based few-shot learning algorithm), MetaOptNet-RR (meta-learning algorithm with differentiable convex optimization), MTL (meta-transfer few-shot learning algorithm), RFS-distill (meta-learning algorithm based on embedding model), and SKD-GEN1 (a few-shot learning method based on fine-tuning). The experimental results are shown in Table 1.

[0125] Table 1 shows the experimental results on the MiniImageNet dataset.

[0126]

[0127] As shown in Table 1, the method of this invention achieves good few-shot image classification results on the MiniImageNet dataset. Compared with the RelationNet and CovaMNet methods, the accuracy is significantly improved. In the experiment with 1 sample per class of 5 images, the accuracy of this method is higher than other few-shot learning methods. Compared with the second-best SKD-GEN1 method, the accuracy is improved by 12.5%.

[0128] The method of this invention was compared with the following algorithms in a small-sample fine-grained image classification experiment on the CUB-200, StanfordDogs, and StanfordCars fine-grained datasets: Deep Nearest Neighbor Neural Network (DN4), Meta-learning Algorithm Based on Embedding Model (RFS-distill), ProtoNet, Graph Neural Network (GNN), Metric Learning-based Small-Shot Learning Algorithm (CovaMNet), Meta-learning-based Small-Shot Learning Algorithm (MAML), MatchingNet, RelationNet, and Local Descriptor-based Multi-Prototype Network Algorithm (LMPNet). The experimental results are shown in Table 2.

[0129] Table 2 shows the experimental results on the fine-grained dataset.

[0130]

[0131] As shown in Table 2, the present invention achieves excellent few-shot image classification results on the CUB-200, StanfordDogs, and StanfordCars fine-grained datasets. The classification accuracy of the present method is higher than that of other few-shot learning methods. In the experiment with 1 sample per class of 5 images, the present invention improves the accuracy by 4.3%, 3.58%, and 1.97% respectively compared to the LMPNet algorithm, which is the second best on the CUB-200, StanfordDogs, and StanfordCars datasets.

[0132] The method of this invention was compared with the few-sample learning algorithm based on meta-learning (MAML), the relation network algorithm (RelationNet), and the graph neural network algorithm (GNN) on the CIIP-TPID dataset. The results are shown in Table 3.

[0133] Table 3 shows the experimental results on the CIIP-TPID dataset.

[0134]

[0135] As shown in Table 3, the present invention achieves good small-sample image classification results on the CIIP-TPID fine-grained dataset. In the experiment with 1 sample per class of 5 image types, the method of the present invention improves performance by 12.32%, 11.75%, and 12.62% respectively compared to the suboptimal GNN algorithm on the surface image set, impression image set, and mixed image set of the CIIP-TPID fine-grained dataset.

Claims

1. A small sample fine-grained image classification method based on a dual metric network, characterized by The following steps consist of: (1) Constructing a small sample data set The public data sets CUB-200, StanfordDogs, StanfordCars and the self-built data set CIIP-TPID are used as experimental data sets. The image size of the CUB-200, StanfordDogs and StanfordCars data sets is processed to be 84x84 pixels, and the image size of the CIIP-TPID data set is processed to be 48x48 pixels. The data sets are divided into training set, validation set and test set according to the ratio of 8:1:

1. The image classes in the training set, validation set and test set do not cross each other, and the images in each class are different from each other. In the training set, validation set and test set, W image classes are randomly selected, K+X samples are randomly selected from the W classes, X is a positive integer not equal to 0, K samples are used as the support set S, and X samples are used as the query set Q. The support set S and the query set Q are of the same class and have different samples within the class. One support set S and one query set Q form one classification unit. The training set, validation set and test set are divided into R classification units. W, K, X, S, Q and R are finite positive integers. (2) Constructing a dual metric network The dual metric network is composed of a feature embedding module and a space-channel attention module. The output ends of the space-channel attention module are connected to the global feature metric network and the local feature metric network to form a dual metric network. The feature embedding module is composed of a convolution layer 1 and 4-12 residual blocks, an average pooling layer 1, a batch normalization layer 1 and a full connection layer 1 connected in sequence. (3) Training the dual metric network 1) Determine the objective function The objective function includes a triplet loss function L1, a cross-entropy loss function L2 and an evaluation function ACC. The triplet loss function L1 is determined as follows: Wherein, a represents a query sample, p represents a sample belonging to a query set category, n represents a sample different from the query set category, represents different samples in the feature embedding module, N is the total number of samples, and N is a finite positive integer; The cross-entropy loss function L2 is determined as follows: wherein y i is the true label of the sample, p i is the probability that the network model predicts that the sample belongs to the i-th class; The evaluation function ACC is determined as follows: Where TP represents the number of positive example predictions, FN represents the number of positive example prediction errors, TN represents the number of negative example prediction errors, FP represents the number of negative example prediction errors, TP and FP, TN, FN are the total number of samples, TP and TN are the number of correctly predicted samples, FN and FP are the number of incorrectly predicted samples, TP, FP, TN, FN∈M, M is a finite positive integer, and TP, FP, TM, FN are not equal to 0 at the same time. 2) Train the dual metric network The support set S and the query set Q in the training set are input into the dual metric network for training. The training process uses the Adam optimizer with a learning rate initialization of 0.

001. The learning rate is reduced by half every 30 rounds of training. The early termination method is used to monitor the overfitting phenomenon in the training process of the dual metric network. (4) Save the model During the training of the dual metric network, the weights are constantly updated, and the weight file is saved. (5) Verify the dual metric network The support set S and the query set Q in the validation set are input into the trained dual metric network for verification. (6) Test the dual metric network The support set S and the query set Q in the test set are input into the trained dual metric network for testing, a saved weight file is loaded, and the classification of the image is obtained.

2. The small sample fine-grained image classification method based on a dual metric network according to claim 1, characterized in that: In the step (2) of constructing the dual metric network, the residual block of the feature embedding module is composed of a convolution layer 2, an activation function layer 1, a convolution layer 3 in sequence.

3. The small sample fine-grained image classification method based on a dual metric network according to claim 1, characterized in that: In the step (2) of constructing the dual metric network, the spatial-channel attention module is composed of a position attention module and a channel attention module in parallel, and a convolution layer 4 in sequence.

4. The small sample fine-grained image classification method based on a dual metric network according to claim 1, characterized in that: In the step (2) of constructing the dual metric network, the global feature metric network is composed of an embedding module, a correlation module, a full connection layer 2, a full connection layer 3, and a Sigmoid function layer in sequence.

5. The small sample fine-grained image classification method based on a dual-metric network according to claim 1, characterized in that: In the step (2) of constructing the dual metric network, the local feature metric network is composed of a convolution embedding module and a covariance metric module in sequence.

6. The small sample fine-grained image classification method based on a dual-metric network according to claim 4, characterized in that: In the step (2) of constructing the dual metric network, the embedding module of the global feature metric network is composed of four convolution layers in sequence; and the correlation module of the global feature metric network is composed of a convolution layer 6, a max pooling layer 1, a convolution layer 7, and a max pooling layer 2 in sequence.

7. The small sample fine-grained image classification method based on a dual-metric network according to claim 5, characterized in that: In the step (2) of constructing the dual metric network, the convolution embedding module of the local feature metric network is composed of a convolution layer 8, a normalization layer 1, an activation function layer 2, a max pooling layer 3, a convolution layer 9, a normalization layer 2, an activation function layer 3, a max pooling layer 4, a convolution layer 10, a normalization layer 3, an activation function layer 4, a convolution layer 11, a normalization layer 4, and an activation function layer 5 in sequence.

8. The small sample fine-grained image classification method based on a dual-metric network according to claim 1, characterized in that: In the step (1) of constructing the small sample data set, in the training set, the validation set and the test set, W image categories are randomly selected, W is 5 or 10, K+X samples are randomly selected from the W categories, K is 1 or 5, X is a positive integer not equal to 0, K samples are used as the support set S, X samples are used as the query set Q, the support set S and the query set Q are of the same category, the samples in the same category are different, one support set S and one query set Q form a classification unit, the training set, the validation set and the test set are divided into R classification units, X, S, Q and R are finite positive integers.

Images