Double-flow small sample fine-grained image classification method and device based on attention mechanism

Abstract

The embodiment of the invention provides a double-flow small sample fine-grained image classification method and device based on an attention mechanism. The method comprises the following steps: forming a task by an image to be queried and a specified number of categories of a support sample set; inputting the task into a trained double-flow model, and obtaining a prediction probability that a current query image belongs to a specified number of categories in a support sample set through first flow processing of the double-flow model; traversing all categories of the support sample set in sequence to obtain a final classification result of the query image; the double-flow model is trained in a task mode by adopting a support sample set and a query sample set in a training set, and the first flow processing process and the second flow processing process of the double-flow model in the training are the same; the input of the second stream is the feature of the source image input by the first stream and a target input image obtained after the attention cutting mechanism is carried out on the source image. According to the method, the problems that the existing small sample fine-grained image classification cannot realize accurate classification and the cost is high are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a dual-stream small sample fine-grained image classification method based on an attention mechanism, a dual-stream small sample fine-grained image classification device based on an attention mechanism, and an electronic device. Background Art

[0002] Image classification technology is an important branch of computer vision, and its purpose is to enable computers to correctly classify image targets with different labels. Fine-grained image classification can be regarded as a sub-branch of image classification, and its characteristic is that the differences between target categories are extremely small, such as Boeing and Airbus, BYD and Tesla, etc. In addition, due to the large number of real categories, it is impossible to guarantee that a large number of samples can be obtained for targets of various categories. Therefore, in order to adapt to the real environment, small sample learning is also one of the important research directions in the field of image classification.

[0003] However, since the image features to be processed for fine-grained image classification have high intra-class variance and low inter-class variance, it is difficult for the model to learn effective discriminative features from complex and subtle appearance changes. Therefore, researchers began to explore how to intuitively use such small differences. Some recent research works, such as the feature information encoding model represented by Bilinear CNNs, learn fine-grained information useful for classification from high-dimensional feature vectors, but the model has extremely high requirements for hardware devices and is prone to dimensionality explosion; in order to enhance the computer's "vision" ability, the attention mechanism is usually used in the feature extraction process to enable the model to extract more fine-grained information. The attention model represented by Attention CNNs generally adopts spatial or channel interaction to reduce the feature dimension that needs to be calculated, while making the model more focused on the target feature. With the popularization of deep learning technology, end-to-end classification methods are a mainstream training framework to reduce model complexity and improve learning efficiency. Some other works focus on semi-supervised and unsupervised learning methods to deal with the difficulty and cost of annotating fine-grained classification samples. Summary of the invention

[0004] The embodiment of the present invention provides a dual-stream small sample fine-grained image classification method and electronic device based on the attention mechanism to solve the problem that small sample fine-grained image classification in the prior art cannot achieve accurate classification and is costly.

[0005] The embodiment of the present invention discloses a two-stream small sample fine-grained image classification method based on an attention mechanism, comprising:

[0006] The query image and the specified number of categories of the support sample set are combined into a task;

[0007] The task is input into the trained two-stream model. After being processed by the first stream of the two-stream model, the predicted probability that the current query image belongs to the specified number of categories in the support sample set is obtained. All categories of the support sample set are traversed in sequence to obtain the final classification result of the query image.

[0008] The two-stream model is trained in a task-based manner using a support sample set and a query sample set in a training set. During the training, the first-stream processing and the second-stream processing of the two-stream model are the same, and the input of the second stream is the target input image obtained after the features of the source image input by the first stream and the source image are subjected to an attention cropping mechanism. During the training, the model parameters are optimized according to the cosine metric loss, cross entropy loss, and KL divergence loss output after the first stream processing and the second stream processing, respectively.

[0009] Optionally, the method further comprises:

[0010] Acquire a training set including a support sample set and a query sample set, and form multiple tasks for training two-stream models according to the support sample set and the query sample set; the training set is a small sample fine-grained image sample set;

[0011] Each task is input into the two-stream model for processing. The first stream processing and the second stream processing of the two-stream model are the same. The attention map output by the first stream is calculated through a linear classifier to calculate the cross entropy loss; the attention map output by the second stream is calculated through a linear classifier to calculate the cross entropy loss;

[0012] and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module, and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module;

[0013] The KL divergence is used to optimize the loss generated by the cosine metric of the first stream and the loss generated by the cosine metric of the second stream;

[0014] The parameters of the two-stream model are optimized according to the cross entropy loss, cosine metric loss and KL divergence optimization loss output by the first stream and the second stream respectively, to obtain the trained two-stream model.

[0015] Optionally, each task is fed into a two-stream model for processing, including:

[0016] The task includes: n images of n categories in the support sample set, 1 query image in the query sample set,

[0017] Input n+1 images into the encoding component of the first stream, extract the features Fq of the query sample and the features Fs of the support sample;

[0018] Process Fs and n category images through the attention cropping mechanism to obtain n new target input images of the support samples, and process Fq and the query image through the attention cropping mechanism to obtain a new target input image of the query image;

[0019] Input all new target input images into the encoding component of the second stream, extract the features Fq-r of the query sample and the features Fs-r of the support sample;

[0020] Fq and n Fs are clustered through the attention mechanism; and Fq-r and n Fs-r are clustered through the attention mechanism.

[0021] Optionally, Fq and n Fs are clustered through an attention mechanism, including:

[0022] The n Fs are linearly learned through a multi-layer perceptron, outputting a class weight for each supporting sample class;

[0023] Each class weight is fused with Fq, that is, each class weight is dot-multiplied with Fq along the channel direction, and the n Fq's after fusion of support sample information are passed through a new multi-layer perceptron; finally, the attention map corresponding to the number of supported sample categories is output; all attention maps are globally pooled and input into the linear classifier and metric classification module.

[0024] Optionally, optimizing the parameters of the two-stream model according to the cross entropy loss, the cosine metric loss, and the KL divergence optimization loss output by the first stream and the second stream respectively includes:

[0025]

[0026] L total =α·(L′ ce +L′ m +L′ KL )+β·(L″ ce +L″ m +L″ KL )

[0027] L ce , L m and L KL They represent the cross entropy loss, the cosine metric loss, and the KL divergence optimization loss respectively; Indicates the prediction score that the nth query sample belongs to the i-th category; L total represents the total loss; N represents the total number of samples in the task, Indicates the prediction score that the nth sample belongs to the jth class;

[0028] K represents the number of categories in the support sample set extracted in each task; It indicates the fraction of the jth attention map generated by the samples in the support sample set belonging to the i-th class after the attention mechanism clustering process; y mi represents the predicted score of the current sample belonging to the i-th class after the cosine measurement by the measurement classification module; L m represents the loss caused by the cosine metric, S and Q represent the number of support samples and the number of query samples respectively, and x s represents the feature vector of the final output of the s-th support sample, x q represents the feature vector finally output by the qth query sample; α and β are hyperparameters; L′ ce , L′ m and L′ KL They represent the cross entropy loss of the first-stream output, the loss of the cosine metric, and the loss of KL divergence optimization, respectively. ce , L″ m , L″ KL represents the cross entropy loss of the second stream output, the loss of the cosine metric, and the loss of the KL divergence optimization.

[0029] The embodiment of the present invention also discloses a two-stream model training method for small sample fine-grained image classification, comprising:

[0030] Acquire a training set including a support sample set and a query sample set, and form multiple tasks for training two-stream models according to the support sample set and the query sample set; the training set is a small sample fine-grained image sample set;

[0031] Each task is input into the two-stream model for processing. The first stream processing and the second stream processing of the two-stream model are the same. The attention map output by the first stream is calculated through a linear classifier to calculate the cross entropy loss; the attention map output by the second stream is calculated through a linear classifier to calculate the cross entropy loss;

[0032] and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module, and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module;

[0033] The KL divergence is used to optimize the loss generated by the cosine metric of the first stream and the loss generated by the cosine metric of the second stream;

[0034] Optimize the parameters of the two-stream model according to the cross entropy loss, cosine metric loss, and KL divergence optimization loss output by the first stream and the second stream respectively, and obtain the trained two-stream model;

[0035] The input of the second stream is the features of the source image input by the first stream and the target input image obtained after the source image is subjected to the attention cropping mechanism.

[0036] Optionally, each task is fed into a two-stream model for processing, including:

[0037] The task includes: n images of n categories in the support sample set, 1 query image in the query sample set,

[0038] Input n+1 images into the encoding component of the first stream to extract the features Fq of the query sample and Fs of the support sample;

[0039] Process Fs and n category images through the attention cropping mechanism to obtain n new target input images of the support samples, and process Fq and the query image through the attention cropping mechanism to obtain a new target input image of the query image;

[0040] Input all new target input images into the encoding component of the second stream, extract the features Fq-r of the query sample and the features Fs-r of the support sample;

[0041] Fq and n Fs are clustered through the attention mechanism; and Fq-r and n Fs-r are clustered through the attention mechanism.

[0042] Optionally, Fq and n Fs are clustered through an attention mechanism, including:

[0043] The n Fs are linearly learned through a multi-layer perceptron, outputting a class weight for each supporting sample class;

[0044] Each class weight is fused with Fq, that is, each class weight is dot-multiplied with Fq along the channel direction, and the n Fq's after fusion of support sample information are passed through a new multi-layer perceptron; finally, the attention map corresponding to the number of supported sample categories is output; all attention maps are globally pooled and input into the linear classifier and metric classification module;

[0045] And / or, the architecture of the first stream processing and the architecture of the second stream processing of the dual-stream model are the same, and have the same parameters;

[0046] and / or, the tasks include: 5-way 1-shot, 3-way 1-shot, 1-way 1-shot;

[0047] The encoding component is a 64*64 convolution component.

[0048] The embodiment of the present invention also discloses a dual-stream small sample fine-grained image classification device based on an attention mechanism, comprising:

[0049] A task acquisition module is used to combine the image to be queried and the specified number of categories of the support sample set into a task;

[0050] The model processing module is used to input the task into the trained two-stream model, obtain the predicted probability that the current query image belongs to the specified number of categories in the support sample set through the first-stream processing of the two-stream model; traverse all categories of the support sample set in sequence to obtain the final classification result of the query image;

[0051] The two-stream model is trained in a task-based manner using a support sample set and a query sample set in a training set. During the training, the first-stream processing and the second-stream processing of the two-stream model are the same, and the input of the second stream is the target input image obtained after the features of the source image input by the first stream and the source image are subjected to an attention cropping mechanism. During the training, the model parameters are optimized according to the cosine metric loss, cross entropy loss, and KL divergence loss output after the first stream processing and the second stream processing, respectively.

[0052] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0053] The memory is used to store computer programs;

[0054] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0055] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0056] The embodiments of the present invention include the following advantages:

[0057] The two-stream small sample fine-grained image classification method based on the attention mechanism in an embodiment of the present invention adopts a task-based approach to obtain the input of the two-stream model. In the specific classification processing, the first branch of the two-stream model is used for processing, which better achieves the improvement of the accuracy of fine-grained image classification without increasing the hardware cost.

[0058] The dual-stream model of the embodiment of the present invention adopts a logical dual-branch structure, shares model parameters, and makes full use of existing image information to obtain new target input images, so that the model processing pays more attention to the target feature area. In the training stage, the first result vector output by each branch of the model, i.e., the attention score error, the second result vector, i.e., the metric error, and the third result vector and the fourth result vector are used to calculate the KL divergence, which is combined with the metric error and the linear classifier cross entropy loss. The three are jointly optimized as a whole, so that the trained model focuses more on the significant features of the query sample, increases the feature weight, and facilitates subsequent classification.

[0059] The two-stream model for fine-grained image classification according to the embodiment of the present invention can process complex image features affected by multiple factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flowchart of a two-stream small sample fine-grained image classification method based on an attention mechanism provided in an embodiment of the present invention;

[0061] Figure 2 It is a flowchart of the steps of a two-stream small sample fine-grained image classification method based on an attention mechanism provided in an embodiment of the present invention;

[0062] Figure 3 is a flowchart of a two-stream model training method for small sample fine-grained image classification provided in an embodiment of the present invention;

[0063] Figure 4 It is a schematic diagram of a process of clustering processing based on an attention mechanism provided in an embodiment of the present invention;

[0064] Figure 5 It is a structural block diagram of a two-stream small sample fine-grained image classification device based on an attention mechanism provided in an embodiment of the present invention;

[0065] Figure 6 is a schematic diagram of a small sample provided in an embodiment of the present invention;

[0066] Figure 7 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0067] Figure 8 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] In order to better understand the concept of small sample classification, the supporting sample set and the query sample set are explained in the embodiments of the present invention.

[0070] Few-shot learning is a machine learning paradigm that aims to enable models to quickly learn new concepts from a small number of samples. In the few-shot classification task, two main concepts are often mentioned: support set and query set. These two sets are crucial to understanding and evaluating the performance of the model under few-shot conditions. Support Set: The support set contains a small number of labeled samples to represent the basic information of each category. Each category in the support set usually contains only a very limited number of examples, such as 1 or 5 samples. The purpose of the support set is to provide the model with enough information to understand the basic characteristics of each category, so that the model can correctly classify new unseen samples based on these limited data. Support set composition: Generally composed of N different categories, each category has K samples (N-way K-shot setting). For example, in a 5-way 1-shot task, the support set will contain 1 sample from each of the 5 different categories.

[0071] Query Set Definition: The query set also includes samples from the same category, but they are not used directly to train the model. Instead, these samples are used to test whether the model can correctly classify them into the correct category. Purpose of Query Set: By comparing the difference between the model's prediction results for the samples in the query set and their true labels, the learning effect of the model and its generalization ability can be evaluated. Query Set Composition: The query set also contains samples from N categories, but the number of samples in each category may be more than the number of samples in the support set. The samples in the query set are used to measure the performance of the model.

[0072] The embodiment of the present invention relates to small sample learning, and is aimed at small sample learning tasks. First, randomly select C from the known category C1 1i There are m categories, and m target images are selected for each category. Due to the small sample scenario, m is usually 1, 3, 5, etc. These sample images constitute the support sample set S. Under the limited data condition, the network needs to make a category judgment on a new query sample image Q. Assuming that lions and tigers both belong to the category of rare images, then Figure 6 As shown in Figure 1, when there are only two sample pictures of lions and tigers, a new image target category needs to be judged.

[0073] The problem with small sample classification is that there is no massive data to pre-train the network. Under this condition, the training method of small sample models must be different from that of traditional big data models.

[0074] In the embodiment of the present invention, an attention generation mechanism based on support samples is used in the constructed dual-stream model. Therefore, in order to better understand the attention mechanism, the attention mechanism is briefly introduced below.

[0075] Attention mechanism is a key technology in deep learning. It imitates the ability of human visual attention and allows the model to focus on the most important parts of the input data. The basic principle of attention mechanism The core idea of ​​attention mechanism is to select highly relevant information from a large amount of information and ignore unimportant information. When processing long sequences, this mechanism can help the model better understand the contextual relationship, thereby improving the performance of the model. Types of attention mechanism: Dot-Product Attention: Determine the weight by calculating the dot product between the query and the key. Additive Attention / Bahdanau Attention: Use an additional fully connected layer to calculate the degree of match between the query and the key. Scaled Dot-Product Attention: A variant of dot-product attention, which divides the dot product by the square root of (d_k) ((d_k) is the dimension of the key) after calculating the dot product to prevent the problem of gradient vanishing or exploding due to excessive dot product. Multi-Head Attention: Run multiple attention mechanisms at the same time, and then splice the results together, so that the model can focus on information in different subspaces. The attention mechanism greatly improves the model's ability to handle complex dependencies.

[0076] Reference Figure 1 and Figure 2 , shows a flowchart of a two-stream small sample fine-grained image classification method based on an attention mechanism provided in an embodiment of the present invention, which may specifically include the following steps:

[0077] 201. The image to be queried and the specified number of categories of the support sample set are combined into a task.

[0078] 202. Input the task into the trained two-stream model, and obtain the predicted probability that the current query image belongs to the specified number of categories in the support sample set through the first-stream processing of the two-stream model; traverse all categories of the support sample set in sequence to obtain the final classification result of the query image.

[0079] The two-stream model is trained in a task-based manner using a support sample set and a query sample set in a training set. During the training, the first-stream processing and the second-stream processing of the two-stream model are the same. The input of the second stream is the target input image obtained after the features of the source image input by the first stream and the source image are subjected to an attention cropping mechanism. The output after the first stream processing and the output after the second stream processing use the KL divergence optimization method to calculate the loss and perform model optimization.

[0080] During training, the model parameters are optimized according to the cosine metric loss, cross entropy loss and KL divergence loss output after the first stream processing and the second stream processing respectively.

[0081] The first-stream processing process is that the image enters SAG for attention clustering processing after passing through the encoding component, and the output attention map then enters the metric classification module to output the predicted probability.

[0082] The dual-stream small-sample fine-grained image classification method of this embodiment fully mines the fine-grained feature area in the small-sample fine-grained image, and uses the attention mechanism to enhance the feature weight of the area, thereby solving the problems of background feature interference and single extracted features in the small-sample fine-grained image classification research.

[0083] Combination Figure 1 As shown in the figure, the query image A and the three categories (two categories are shown in the figure) of the support sample set form a task, namely, a 3-way 1-shot task; the above task is input into the encoding component of the two-stream model. At this time, the query image A outputs the feature map Fa through the encoding component, and the feature map Fs is obtained after the encoding component of B. All feature maps are processed by the attention generation module based on the support sample, as shown in Figure 5 The specific processing process shown is that the output of the attention generation module based on the support sample is processed by the metric classification module to obtain the final classification result, namely the predicted probability.

[0084] correspond Figure 5 In this model processing, all feature maps of the support sample set are linearly learned through the multi-layer perception module (i.e., multi-layer perceptron), and a class weight is output for each support sample class. The class weight and the feature map of the query image A are fused (i.e., each class weight is multiplied with the feature map of the query image A along the channel direction), and then the features of the query image that has been fused with the support sample information are passed through the multi-layer perceptron again to output a set of vectors, i.e., the attention map corresponding to the number of support sample categories. If visualization is required, all attention maps can be output, or the attention map with the largest activation value can be selected as the standard output.

[0085] The support weight of this embodiment may be a pre-given value. The multi-layer perception module may be a module composed of multiple groups of fully connected layers.

[0086] The above method can be used in a two-stream model for fine-grained image classification, which can handle complex image features affected by multiple factors.

[0087] In the specific implementation process, the above Figure 2 Before performing classification, the method of this embodiment further includes a step of training the dual-stream model, which is as follows:

[0088] 200a. Obtain a training set including a support sample set and a query sample set, and form multiple tasks for training two-stream models according to the support sample set and the query sample set; the training set is a small sample fine-grained image sample set;

[0089] 200b. Input each task into the two-stream model for processing. The first stream processing and the second stream processing of the two-stream model are the same. The attention map output by the first stream is calculated through a linear classifier to calculate the cross entropy loss; the attention map output by the second stream is calculated through a linear classifier to calculate the cross entropy loss; and the attention map output by the first stream is output through the metric classification module to output the loss generated by the cosine metric, and the attention map output by the first stream is output through the metric classification module to output the loss generated by the cosine metric; the KL divergence is used to optimize the loss generated by the cosine metric of the first stream and the loss generated by the cosine metric of the second stream; the parameters of the two-stream model are optimized according to the cross entropy loss, cosine metric loss and KL divergence optimized loss output by the first stream and the second stream respectively, to obtain the trained two-stream model.

[0090] Combination Figure 3 and Figure 4 For example, each task is input into the two-stream model for processing, including:

[0091] The task includes: n images of n categories in the support sample set, 1 query image in the query sample set,

[0092] Input n+1 images into the encoding component of the first stream to extract the features Fq of the query sample and Fs of the support sample;

[0093] Process Fs and n category images through the attention cropping mechanism to obtain n new target input images of support samples, and process Fq and the query image through the attention cropping mechanism to obtain a new target input image of the query image;

[0094] Input all new target input images into the encoding component of the second stream, extract the features Fq-r of the query sample and the features Fs-r of the support sample;

[0095] Fq and n Fs are clustered through the attention mechanism; and Fq-r and n Fs-r are clustered through the attention mechanism to obtain the attention maps of their respective outputs.

[0096] In this embodiment, Fq and n Fs are clustered through an attention mechanism, including:

[0097] The n Fs are linearly learned through a multi-layer perceptron, outputting a class weight for each supporting sample class;

[0098] Each class weight is fused with Fq, that is, each class weight is dot-multiplied with Fq along the channel direction, and the n Fq's after fusion of support sample information are passed through a new multi-layer perceptron; finally, the attention map corresponding to the number of supported sample categories is output; all attention maps are globally pooled and input into the linear classifier and metric classification module

[0099] That is, during the training phase, the error function also takes into account the KL divergence of the two-stream model and the cross entropy loss of the linear classifier.

[0100] The linear classifier in this embodiment uses the standard results for comparison during the training process, and the cross entropy loss obtained is the same as the existing comparison, and the larger the value, the better the match;

[0101] The Metric (metric classification module) in this embodiment uses the cosine metric to calculate the similarity between the query sample and the support sample in the high-dimensional feature space for classification. The smaller the value, the better the match.

[0102] In this embodiment, the architecture of the first stream processing and the architecture of the second stream processing of the dual-stream model are the same, and the parameters are the same; and / or, the tasks include: 5-way 1-shot, 3-way 1-shot, 1-way 1-shot; the encoding component is a 64*64 convolution component.

[0103] The two-stream model trained in this embodiment can fully mine the fine-grained feature area in the small sample fine-grained image, and use the attention mechanism to enhance the feature weight of the area, so as to solve the problems of background feature interference and single extracted features in the small sample fine-grained image classification research; the metric learning method is used for the final classification during the training process; at the same time, the KL divergence is introduced to constrain the two-stream metric error and enhance the model learning effect.

[0104] like Figure 3 and Figure 4 As shown, the dual-stream model training method for small sample fine-grained image classification in this embodiment can be described in advance before describing the steps of the training method.

[0105] The two-stream model is an effective model architecture in the field of fine-grained image classification, and its implementation is mainly based on the attention mechanism.

[0106] The first-stream processing of the dual-stream model in this embodiment includes: obtaining a target image after the features extracted by the convolution layer in the traditional convolutional neural network are subjected to an attention cropping mechanism and up-sampled to the original image resolution. The target image will pay more attention to the area where the real target is located and can effectively remove the background area in the original image; that is, after the target image and the original image are processed by attention pruning, part of the background area in the original image is cropped to obtain an output image as a new target input image.

[0107] The output image is input into the second stream of the two-stream model again for feature learning, thus forming a logical "two-stream" model. The two-stream model shares all model parameters, aiming to fully exploit the feature information in the existing image that is helpful for classification, and is equivalent to expanding the input sample set to twice the original size.

[0108] Both the first and second streams in the two-stream model include a Support-guilded Attention Generator module (i.e., an attention generation module based on support samples).

[0109] The general small sample image classification model does not utilize the connection between the support set and query set sample features. In order to utilize the feature information of the support sample during feature extraction, the Support-guildedAttention Generator (SAG) module proposed in this embodiment is used to enhance the feature representation capability of the query sample.

[0110] Taking the 5-way 1-shot task (5 support samples - 1 query image) as an example, 5 categories of support samples are input, and 5 attention maps are generated after the Encoder module and the SAG module. After the attention mechanism clustering processing and visualization, it can be observed that when the input 5 samples belong to the same category as the sample of the query image, the obtained feature activation value is the largest. In the training stage, in order to optimize the generation of attention maps, all attention maps generated based on support samples in this embodiment are first pooled, then input into the linear classifier and optimized using the cross entropy loss, as shown below:

[0111]

[0112] Indicates the prediction score that the nth query sample belongs to the i-th category, Indicates the prediction score that the nth sample belongs to the jth class, Lc e represents the cross entropy loss.

[0113] The attention output of SAG in this embodiment Figure 1On the one hand, a linear classifier is used for processing, that is, this embodiment pools the generated attention map, inputs it and optimizes it using cross entropy loss. On the other hand, KL divergence optimization is used to achieve loss optimization.

[0114] That is, this embodiment uses the metric learning method commonly used in the field of small sample learning for classification, and uses the cosine distance to calculate the classification loss. In the training stage, in order to enhance the global feature learning ability, the cross entropy loss function commonly used in traditional classification tasks is retained, and the KL divergence is used to optimize the metric error.

[0115] KL divergence is usually used to study the distribution of data in industrial process fault diagnosis, and is less used in the field of images. The Support-guilded Attention Generator module and the final metric classification module in this embodiment will produce errors during the training process. In order to consider the distribution of different branches, the KL divergence is tried for optimization.

[0116] Specifically,

[0117]

[0118] L total =α·(L′ ce +L′ m +L′ KL )+β·(L″ ce +L″ m +L″ KL );

[0119] Among them, K represents the number of categories extracted in each training task (i.e. Task); It indicates the fraction of the jth attention map generated by the sample in the support sample set belonging to the i-th class after the SAG module; ym i Indicates the predicted score of the current sample belonging to the i-th class after cosine measurement; L m represents the loss generated by the cosine metric, S and Q represent the number of support samples and the number of query samples respectively, and x s is the feature vector of the final output of the s-th support sample, x q is the feature vector finally output by the qth query sample. α and β are hyperparameters, and in this embodiment, their values ​​are 0.3 and 0.7; L′ ce , L′ m and L′ KL They represent the cross entropy loss of the first-stream output, the loss of the cosine metric, and the loss of KL divergence optimization, respectively. ce , L″ m , L″ KLrepresents the cross entropy loss of the second stream output, the loss of the cosine metric, and the loss of the KL divergence optimization.

[0120] That is to say, in order to make the model pay more attention to the target feature area, during the training phase, the attention score error and the metric error output by each branch of the model are used to calculate the KL divergence. The metric error and the linear classifier cross entropy loss are combined to jointly optimize the model as a whole.

[0121] Compared with the existing fine-grained image classification technology, the method of this embodiment adopts a two-stream model, which can extract more detailed feature information on the one hand, and on the other hand, the two-stream model uses an attention mechanism to generate a sub-image, i.e., a feature map, for each original input image, thereby increasing the number of samples. The use of attention mechanism clustering processing can cross-learn the query sample features and the support sample features, so that the model focuses more on the significant features of the query sample, increases the feature weight, and facilitates subsequent classification.

[0122] In this embodiment, in order to solve the problems of difficulty in finding features and large feature granularity in the field of small sample fine-grained image classification, the data set first selects the public benchmark data set CUB-200-2011 in the field of fine-grained image classification, which consists of 200 bird subclasses. Since the number of samples in this data set is small and the image resolution is different, the number of samples is expanded by using techniques such as horizontal flipping, and color jitter and Gaussian noise are randomly added to the image, which is conducive to the model learning invariant features and enhancing the robustness of the model.

[0123] After image preprocessing, the entire dataset is divided into three parts: training set, test set, and validation set. In the training phase, the two-stream model retains the linear classifier in the traditional model, that is, the fully connected layer output features are used to predict the category score using normalization functions such as softmax or sigmoid. The purpose is to enhance the network's self-learning ability for each image feature. In the testing and validation phases, only the metric module is used to calculate the metric score, and the metric method is the cosine metric.

[0124] The attention mechanism is an effective feature extraction method. It performs different combination calculations and threshold filtering on the high-dimensional features extracted by the convolution layer in the network along the channel dimension and the spatial dimension to obtain or enhance some features that are useful for the final classification. The whole process will increase the weight of significant features and weaken the weight of invalid features. Figure 4As shown, in this embodiment, the convolutional features (i.e., feature maps) extracted by the encoding component (such as a Conv4 backbone network or an Encoder network) are first subjected to linear learning through a multi-layer perceptron to output a class weight for each supporting sample class; then the class weights and query samples are fused, each class weight is dot-multiplied with the query sample along the channel direction, and the query sample features after fusion of the supporting sample information are passed through a new multi-layer perceptron; finally, an attention map corresponding to the number of supporting sample categories is output, and the attention map with the largest activation value is selected as the standard output, which is first globally pooled and then connected to the metric classification module to calculate the attention map error, which is used for joint optimization training.

[0125] In this embodiment, the cosine metric is used to calculate the similarity between the query sample and the support sample in the high-dimensional feature space for classification. In the training phase, the error function also considers the KL divergence of the two-stream model and the cross entropy loss of the linear classifier.

[0126] In the prediction stage, we only need to input the original image into the first-stream branch of the two-stream model for prediction, and no more attention cropping is performed. The prediction result is directly output according to the metric classification module.

[0127] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0128] Reference Figure 5 , shows a structural block diagram of a two-stream small sample fine-grained image classification device based on an attention mechanism provided in an embodiment of the present invention, which may specifically include the following modules:

[0129] The task acquisition module is used to combine the image to be queried and the specified number of categories of the support sample set into a task.

[0130] The model processing module is used to input the task into the trained two-stream model, and obtain the predicted probability that the current query image belongs to the specified number of categories in the support sample set through the first-stream processing of the two-stream model; traverse all categories of the support sample set in sequence to obtain the final classification result of the query image.

[0131] The two-stream model is trained in a task-based manner using a support sample set and a query sample set in a training set. During the training, the first-stream processing and the second-stream processing of the two-stream model are the same, and the input of the second stream is the target input image obtained after the features of the source image input by the first stream and the source image are subjected to an attention cropping mechanism. During the training, the model parameters are optimized according to the cosine metric loss, cross entropy loss, and KL divergence loss of the outputs processed by the first stream and the second stream respectively.

[0132] Specifically, the above device also includes a training module not shown in the figure.

[0133] The training module is specifically used to obtain a training set including a support sample set and a query sample set, and form multiple tasks of training two-stream models according to the support sample set and the query sample set; the training set is a small sample fine-grained image sample set;

[0134] Each task is input into the two-stream model for processing. The first stream processing and the second stream processing of the two-stream model are the same. The attention map output by the first stream is calculated through a linear classifier to calculate the cross entropy loss; the attention map output by the second stream is calculated through a linear classifier to calculate the cross entropy loss;

[0135] and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module, and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module;

[0136] The KL divergence is used to optimize the loss generated by the cosine metric of the first stream and the loss generated by the cosine metric of the second stream;

[0137] Optimize the parameters of the two-stream model according to the cross entropy loss, cosine metric loss, and KL divergence optimization loss output by the first stream and the second stream respectively, and obtain the trained two-stream model;

[0138] The input of the second stream is the features of the source image input by the first stream and the target input image obtained after the source image is subjected to the attention cropping mechanism.

[0139] In addition, the training module is also used to input n+1 images into the encoding component of the first stream to extract the features Fq of the query sample and the features Fs of the support sample;

[0140] Process Fs and n category images through the attention cropping mechanism to obtain n new target input images of the support samples, and process Fq and the query image through the attention cropping mechanism to obtain a new target input image of the query image;

[0141] Input all new target input images into the encoding component of the second stream, extract the features Fq-r of the query sample and the features Fs-r of the support sample;

[0142] Fq and n Fs are clustered through the attention mechanism; and Fq-r and n Fs-r are clustered through the attention mechanism. For example, n Fs are linearly learned through a multi-layer perceptron to output a class weight for each supporting sample class; each class weight is fused with Fq, that is, each class weight is dot-multiplied with Fq along the channel direction, and the n Fq' after fusion of the supporting sample information is passed through a new multi-layer perceptron; finally, an attention map corresponding to the number of supporting sample categories is output; all attention maps are globally pooled and input into the linear classifier and metric classification module.

[0143] The device of this embodiment uses a trained dual-stream model to accurately classify small sample images, pays more attention to target feature areas during the classification process, and better improves the accuracy of fine-grained image classification without increasing hardware costs.

[0144] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0146] Memory 1303, used for storing computer programs;

[0147] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0148] The query image and the specified number of categories of the support sample set are combined into a task;

[0149] The task is input into the trained two-stream model. After being processed by the first stream of the two-stream model, the predicted probability that the current query image belongs to the specified number of categories in the support sample set is obtained. All categories of the support sample set are traversed in sequence to obtain the final classification result of the query image.

[0150] The two-stream model is trained in a task-based manner using a support sample set and a query sample set in a training set. During the training, the first-stream processing and the second-stream processing of the two-stream model are the same, and the input of the second stream is the target input image obtained after the features of the source image input by the first stream and the source image are subjected to an attention cropping mechanism. During the training, the model parameters are optimized according to the cosine metric loss, cross entropy loss, and KL divergence loss of the outputs processed by the first stream and the second stream respectively.

[0151] or,

[0152] Acquire a training set including a support sample set and a query sample set, and form multiple tasks for training two-stream models according to the support sample set and the query sample set; the training set is a small sample fine-grained image sample set;

[0153] Each task is input into the two-stream model for processing. The first stream processing and the second stream processing of the two-stream model are the same. The attention map output by the first stream is calculated through a linear classifier to calculate the cross entropy loss; the attention map output by the second stream is calculated through a linear classifier to calculate the cross entropy loss;

[0154] and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module, and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module;

[0155] The KL divergence is used to optimize the loss generated by the cosine metric of the first stream and the loss generated by the cosine metric of the second stream;

[0156] Optimize the parameters of the two-stream model according to the cross entropy loss, cosine metric loss, and KL divergence optimization loss output by the first stream and the second stream respectively, and obtain the trained two-stream model;

[0157] The input of the second stream is the features of the source image input by the first stream and the target input image obtained after the source image is subjected to the attention cropping mechanism.

[0158] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0159] The communication interface is used for communication between the above terminal and other devices.

[0160] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0161] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0162] like Figure 8 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, which stores instructions. When the computer-readable storage medium is run on a computer, the computer executes the two-stream small sample fine-grained image classification method based on the attention mechanism described in the above embodiment.

[0163] In another embodiment provided by the present invention, a computer program product containing instructions is also provided. When the computer is run on a computer, the computer executes the two-stream small sample fine-grained image classification method based on the attention mechanism described in the above embodiment.

[0164] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0165] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0166] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A two-stream small sample fine-grained image classification method based on attention mechanism, characterized in that: include: The query image and the specified number of categories of the support sample set are combined into a task; Input the task into the trained two-stream model, and obtain the predicted probability that the current query image belongs to the specified number of categories in the support sample set through the first-stream processing of the two-stream model; Traverse all categories of the support sample set in sequence to obtain the final classification result of the query image; The two-stream model is trained in a task-based manner using a support sample set and a query sample set in a training set. During the training, the first stream processing and the second stream processing of the two-stream model are the same. The input of the second stream is the target input image obtained by performing an attention cropping mechanism on the source image of the first stream input and the source image. During training, the model parameters are optimized according to the cosine metric loss, cross entropy loss and KL divergence loss output after the first stream processing and the second stream processing respectively.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a training set including a support sample set and a query sample set, and form multiple tasks for training two-stream models according to the support sample set and the query sample set; the training set is a small sample fine-grained image sample set; Each task is input into the two-stream model for processing. The first stream processing and the second stream processing of the two-stream model are the same. The attention map output by the first stream is calculated through a linear classifier to calculate the cross entropy loss; the attention map output by the second stream is calculated through a linear classifier to calculate the cross entropy loss; and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module, and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module; The KL divergence is used to optimize the loss generated by the cosine metric of the first stream and the loss generated by the cosine metric of the second stream; The parameters of the two-stream model are optimized according to the cross entropy loss, cosine metric loss and KL divergence optimization loss output by the first stream and the second stream respectively, to obtain the trained two-stream model.

3. The method according to claim 2, characterized in that Each task is input into the two-stream model for processing, including: The task includes: n images of n categories in the support sample set, 1 query image in the query sample set, Input n+1 images into the encoding component of the first stream, extract the features Fq of the query sample and the features Fs of the support sample; Process Fs and n category images through the attention cropping mechanism to obtain n new target input images of the support samples, and process Fq and the query image through the attention cropping mechanism to obtain a new target input image of the query image; Input all new target input images into the encoding component of the second stream, extract the features Fq-r of the query sample and the features Fs-r of the support sample; Fq and n Fs are clustered through the attention mechanism; and Fq-r and n Fs-r are clustered through the attention mechanism.

4. The method according to claim 3, characterized in that Fq and n Fs are clustered through the attention mechanism, including: The n Fs are linearly learned through a multi-layer perceptron, outputting a class weight for each supporting sample class; Each class weight is fused with Fq, that is, each class weight is dot-multiplied with Fq along the channel direction, and the n Fq's after fusion of support sample information are passed through a new multi-layer perceptron; finally, the attention map corresponding to the number of supported sample categories is output; all attention maps are globally pooled and input into the linear classifier and metric classification module.

5. The method according to claim 3, characterized in that: The parameters of the two-stream model are optimized according to the cross entropy loss, cosine metric loss, and KL divergence optimization loss output by the first and second streams, including: L total =α·(L′ ce +L′ m +L′ KL )+β·(L″ ce +L″ m +L″ KL ) L ce , L m and L KL They represent the cross entropy loss, the cosine metric loss, and the KL divergence optimization loss respectively; y i n Indicates the prediction score that the nth query sample belongs to the i-th category; L total represents the total loss; N represents the total number of samples in the task, y j n Indicates the prediction score that the nth sample belongs to the jth class; K represents the number of categories in the support sample set extracted in each task; It indicates the fraction of the jth attention map generated by the samples in the support sample set belonging to the i-th class after the attention mechanism clustering process; y mi represents the predicted score of the current sample belonging to the i-th class after the cosine measurement by the measurement classification module; L m represents the loss caused by the cosine metric, S and Q represent the number of support samples and the number of query samples respectively, and x s represents the feature vector of the final output of the s-th support sample, x q represents the feature vector finally output by the qth query sample; α and β are hyperparameters; L′ ce , L′ m and L′ KL They represent the cross entropy loss of the first-stream output, the loss of the cosine metric, and the loss of KL divergence optimization, respectively. ce , L″ m , L″ KL represents the cross entropy loss of the second stream output, the loss of the cosine metric, and the loss of the KL divergence optimization.

6. A two-stream model training method for small sample fine-grained image classification, characterized in that: include: Obtain a training set including a support sample set and a query sample set, and form multiple tasks for training two-stream models according to the support sample set and the query sample set; The training set is a small sample fine-grained image sample set; Each task is input into the two-stream model for processing. The first stream processing and the second stream processing of the two-stream model are the same. The attention map output by the first stream is calculated through a linear classifier to calculate the cross entropy loss; the attention map output by the second stream is calculated through a linear classifier to calculate the cross entropy loss; and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module, and the attention map output by the first stream is passed through the loss generated by the cosine metric output by the metric classification module; The KL divergence is used to optimize the loss generated by the cosine metric of the first stream and the loss generated by the cosine metric of the second stream; Optimize the parameters of the two-stream model according to the cross entropy loss, cosine metric loss, and KL divergence optimization loss output by the first stream and the second stream respectively, and obtain the trained two-stream model; The input of the second stream is the features of the source image input by the first stream and the target input image obtained after the source image is subjected to the attention cropping mechanism.

7. The method according to claim 6, characterized in that Each task is input into the two-stream model for processing, including: The task includes: n images of n categories in the support sample set, 1 query image in the query sample set, Input n+1 images into the encoding component of the first stream to extract the features Fq of the query sample and Fs of the support sample; Process Fs and n category images through the attention cropping mechanism to obtain n new target input images of the support samples, and process Fq and the query image through the attention cropping mechanism to obtain a new target input image of the query image; Input all new target input images into the encoding component of the second stream, extract the features Fq-r of the query sample and the features Fs-r of the support sample; Fq and n Fs are clustered through the attention mechanism; and Fq-r and n Fs-r are clustered through the attention mechanism.

8. The method according to claim 7, characterized in that Fq and n Fs are clustered through the attention mechanism, including: The n Fs are linearly learned through a multi-layer perceptron, outputting a class weight for each supporting sample class; Each class weight is fused with Fq, that is, each class weight is dot-multiplied with Fq along the channel direction, and the n Fq's after fusion of support sample information are passed through a new multi-layer perceptron; finally, the attention map corresponding to the number of supported sample categories is output; all attention maps are globally pooled and input into the linear classifier and metric classification module; And / or, the architecture of the first stream processing and the architecture of the second stream processing of the dual-stream model are the same, and have the same parameters; and / or, the tasks include: 5-way 1-shot, 3-way 1-shot, 1-way 1-shot; The encoding component is a 64*64 convolution component.

9. A two-stream small sample fine-grained image classification device based on attention mechanism, characterized in that: include: A task acquisition module is used to combine the image to be queried and the specified number of categories of the support sample set into a task; A model processing module is used to input the task into the trained two-stream model, and obtain the predicted probability that the current query image belongs to a specified number of categories in the support sample set through the first stream processing of the two-stream model; Traverse all categories of the support sample set in sequence to obtain the final classification result of the query image; The two-stream model is trained in a task-based manner using a support sample set and a query sample set in a training set. During the training, the first stream processing and the second stream processing of the two-stream model are the same. The input of the second stream is the target input image obtained by performing an attention cropping mechanism on the source image of the first stream input and the source image. During training, the model parameters are optimized according to the cosine metric loss, cross entropy loss and KL divergence loss output after the first stream processing and the second stream processing respectively.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 8 when executing the program stored in the memory.

Images