An attribute and attention guide-based small sample image classification method

Abstract

The application discloses a small sample image classification method based on attributes and attention guidance, constructs a network framework on the basis of small sample data conditions and a metric learning method, and comprises four parts: a self-attention network, an inter-attention network, a self-adaptive fusion network and an attribute prediction network. The self-attention network is used for extracting image features; the inter-attention network captures the regional correlation between images; the self-adaptive fusion network fuses visual features and attribute information, and realizes correction of image feature embedding; and the attribute prediction network positions attribute features and performs prediction in a weakly supervised manner with the aid of attribute information. Through the use of the attention mechanism and the assistance of the attribute information, more discriminative features are collaboratively generated, a more general model is obtained, and the image classification precision is finally improved. The method is simple in program and easy to implement; the system is easy to construct, and the image classification under the small sample condition can be completed in an end-to-end mode with a deep neural network.

Description

Technical Field

[0001] This invention belongs to the field of computer vision image classification, and relates to feature extraction and classification of image samples under conditions of small sample data using attention networks and attribute prediction networks. Specifically, it relates to a small sample image classification method based on attributes and attention guidance. Background Technology

[0002] Few-sample image classification has become an important topic in the field of computer vision. The goal of this problem is to improve the accuracy of image classification under the condition of a very small number of samples. The challenge lies in improving the generalization ability of the model and preventing network overfitting.

[0003] Existing few-shot learning methods train neural networks to embed support and query samples into the feature space and distinguish them using distance metrics. However, these methods still have room for exploration in improving the utilization of image information. Furthermore, existing methods lack research on the relationship between support and query set images, leading to low recognition accuracy. Some few-shot methods utilize auxiliary semantic information to enhance feature representations to compensate for the lack of supervision and improve performance. Following a realistic context, these methods all assume that the support set can acquire semantic information, while the query set, as the prediction target, cannot. However, focusing solely on how to learn support sample class centers through multimodal information mixing ignores the unique characteristic of query samples lacking semantic information. This leads to information asymmetry and potential performance loss. Moreover, due to the heterogeneous structure of visual and semantic feature spaces, query set sample features using only visual information may introduce cross-modal biases with support set sample class centers that mix visual and semantic information, thereby reducing recognition accuracy. Summary of the Invention

[0004] This invention aims to overcome the shortcomings of existing technologies and provides a few-sample image classification framework based on attributes and attention guidance. Utilizing attention mechanisms and attribute semantic information, it generates a more generalizable classification model, improving recognition accuracy. The framework consists of four parts: a self-attention network, an attribute prediction network, an adaptive fusion network, and a mutual attention network. Based on metric learning methods, this invention designs a self-attention network to extract image features using a self-attention mechanism. Furthermore, this invention proposes an attribute prediction network that can use attribute semantic auxiliary information to locate and predict attribute features through weak supervision, thereby obtaining the attribute features of the samples. An adaptive fusion network fuses visual features with attribute information to achieve corrected image feature embedding. A mutual attention network is used to focus on regional correlations between images, generating more discriminative features and obtaining a more generalizable model with limited sample data.

[0005] The specific technical solution of this invention is as follows:

[0006] A few-sample image classification method based on attributes and attention guidance includes the following steps:

[0007] 1) Prepare initial data: Initial data includes image data, category labels, and attribute labels.

[0008] 2) Construction of self-attention network: Image samples are input into ResNet-12 network to obtain primary visual features, and then the primary visual features of the samples are converted into self-attention features through attention mechanism to provide reliable input for the next module.

[0009] 3) Construction of the attribute prediction network: Input the query sample image into the channel attention network to obtain attention features, and then predict the attributes contained in the current sample through the attribute localization module.

[0010] Image features are input into a channel attention network to obtain attention features, which are then input into the attribute localization module. The attribute localization module obtains the estimated transformation matrix corresponding to the attention features through a fully connected layer. Based on the estimated transformation matrix, the attention features are sampled by bilinear interpolation. The sampled features are then passed through a fully connected layer to obtain the final attribute prediction features.

[0011] 4) Construction of adaptive fusion network: The self-attention features output in step 2) are adaptively fused with the attribute information in step 3) to obtain the corrected image features.

[0012] The query sample attribute prediction features and support sample attribute label vectors are input into the weight calculation network to obtain their respective fusion ratio parameters λ1 and λ2. The weight calculation network consists of two linear layers. The adaptive fusion network fuses the input query sample self-attention features with the query sample prediction features at a ratio of λ1, and fuses the support sample self-attention features with the support sample label vectors at a ratio of λ2, resulting in a fused feature that combines visual features and attribute information.

[0013] 5) Construction of mutual attention network: Analyze the regional correlation between images based on the fusion features of the support samples and query samples obtained in step 4).

[0014] 5-1) Input a pair of fused features for support samples and query samples. First, use a point convolutional layer to transform them into more compact features. The mutual attention calculation process constructs a correlation tensor based on cosine similarity.

[0015] 5-2) The relevant tensors are input into a convolutional block for geometric matching. This convolutional block consists of two four-dimensional convolutional layers. The first convolutional layer generates multiple relevant tensors, and the second convolutional layer aggregates these tensors into a single relevant tensor, thus obtaining the mutual attention relevant tensor, which represents the matching score between samples. This process achieves geometric matching by analyzing the consistency of adjacent matches in four-dimensional space.

[0016] 5-3) Mutual Attention Map Calculation. The mutual attention correlation tensor is converted into a mutual attention map. The calculation of the mutual attention map converts the matching score between samples into the average matching probability between each position in one sample and all positions in another sample. Multiplying each position of the fused feature by the mutual attention map yields the final feature. The formula for calculating the final feature of the query sample is as follows:

[0017]

[0018] Where, q∈R C To query the final features of the sample, x q Att represents the spatial location on the feature map of the queried sample. q ∈R H×W To query the mutual attention mapping graph of the samples, To query the fused features that combine visual features and attribute information of a sample, ∑ represents the summation operator.

[0019] The formula for calculating the final features of the supporting samples is as follows:

[0020]

[0021] Where, s∈R C To support the final features of the sample, x s Att represents the spatial location on the feature map of the supporting sample. s ∈R H×W To support the mutual attention mapping of samples, To support fusion features that combine visual features and attribute information of samples, ∑ represents the summation operator.

[0022] 6) Train the convolutional neural network obtained from steps 2), 3), 4), and 5). The supervised training method uses the constructed loss function to iteratively optimize the network parameters. Once training is complete, test the trained model on a test set to obtain the corresponding image classification results.

[0023] 6-1) Compute L using an additional fully connected classification layer on the base network ResNet-12. anchor The loss guides the model classification, and the calculation formula is as follows:

[0024]

[0025] Z q This represents the primary visual features of the query sample, where W and b are the weights and biases in the fully connected layer, respectively, and c∈C. train Indicates the training set category.

[0026] 6-2) For the network framework, including self-attention networks, attribute prediction networks, adaptive fusion networks, and mutual attention networks, end-to-end supervised training is performed as a whole. The metric-based loss-guided model maps query features to prototypes of similar classes, and calculates the metric-based loss L using the cosine similarity between the query sample and the support set prototypes. metric :

[0027]

[0028] Where sim(,) is the cosine similarity and τ is the scalar coefficient. This represents the average of the feature vectors of K query samples from n classes. Similarly, Let be a set of prototypes, and let be the average of the feature vectors of the K support samples of n classes.

[0029] 6-3) For the attribute prediction network part, a weighted binary cross-entropy loss function L is also used. attribute Supervision is carried out using attribute information as labels.

[0030]

[0031] Where M is the number of attributes, a m It is the tag with the m-th attribute. σ is the predicted value of the m-th attribute, and σ represents the sigmoid activation function.

[0032] 6-4) Finally, combining the definitions of all loss functions and scaling parameters, the following is a summary:

[0033] L = L anchor +vL metric +μL attribute

[0034] Where ν and μ are hyperparameters of the balancing loss term.

[0035] The beneficial effects of this invention are:

[0036] This invention, based on deep neural networks, constructs a few-sample image classification network guided by attributes and attention. Given a small number of images, it employs an attention mechanism to extract features and focuses on regional correlations between samples. It uses predicted target attribute information to assist visual information in accurately generating target class centers. This method has the following characteristics:

[0037] 1. The system is easy to build; it can use deep neural networks to support end-to-end image input and obtain the final classification result.

[0038] 2. This method assists in few-sample target recognition tasks through attribute prediction. It generates attribute semantics and corrects image feature embeddings, effectively alleviating the information asymmetry problem between support samples and query samples.

[0039] 3. This method uses a self-attention mechanism to extract feature representations, and with the assistance of attribute information, it fully explores the relationship between the support set and the query set at the image level using a mutual attention mechanism to generate more discriminative class centers and improve generalization.

[0040] 4. The program is simple and easy to implement, and the algorithm runs fast. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method of the present invention.

[0042] Figure 2 This is the overall network structure diagram of the method of the present invention.

[0043] Figure 3 This is a diagram of a self-attention network structure.

[0044] Figure 4 This is a diagram of the mutual attention network structure. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0046] This invention provides a few-sample image classification method based on attributes and attention guidance, the specific implementation process of which is as follows: Figure 1 As shown, the overall network structure is as follows Figure 2 As shown, the method in this embodiment includes the following steps:

[0047] 1) Prepare initial data: The initial data is the CUB (C.Wah, S.Branson, P.Welinder, P.Perona, and S.Belongie. The Caltech-UCSD Birds-200-2011 Dataset. California Institute of Technology, 2011.) public dataset, including image data, category labels and attribute labels.

[0048] 2) Construction of self-attention networks, such as Figure 3As shown: Image samples are input into the base network to obtain primary visual features, and then the primary visual features of the samples are converted into self-attention features through an attention mechanism, providing reliable input for the next module.

[0049] Primary visual features Z∈R were extracted using a ResNet-12 network (He K, Zhang X, Ren S, et al. Deep Residual Learning for Image Recognition[J].IEEE, 2016.). H×W×C .

[0050] During the self-attention computation, the Hadamard products of the feature H×W tensors of dimension C with their neighborhood vectors are calculated, and these are then aggregated into a self-attention tensor N∈R. H×W×U×V×C Zero-padding was applied to the edges of the features to facilitate sampling. In this embodiment, the neighborhood sampling ranges U and V are both set to 5.

[0051] The convolutional block g(·) reduces the spatial dimension of the self-attention tensor N through two-dimensional convolution. The output feature g(N)∈R... H ×W×C Adding this to the feature Z yields the final self-attention feature F∈R. H×W×C

[0052] 3) Construction of the attribute prediction network, such as Figure 2 The attribute prediction network section shows that the query sample image is input into the channel attention network to obtain attention features, and then the attribute localization module predicts the attributes contained in the current sample.

[0053] 3-1) First, the basic features of the query sample image are extracted by the BN-Inception network (Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448–456, 2015.).

[0054] 3-2) The basic features are transformed into attention features through the channel attention network SE-Net (Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7132–7141, 2018.).

[0055] 3-3) Then, the attention features are input into the attribute localization module. The attribute localization module obtains the estimated transformation matrix corresponding to the attention features through a fully connected layer, uses bilinear interpolation sampling to obtain region-based features, and finally performs attribute classification prediction. Thus, the query sample image is transformed into query sample attribute prediction features.

[0056] 4) Construction of adaptive fusion network: Adaptively fuse the self-attention features output in step 2) with the attribute information output in step 3).

[0057] By inputting the query sample attribute prediction features and support sample attribute label vectors into the weight calculation network, their respective fusion ratio parameters λ1 and λ2 can be obtained. The weight calculation network consists of two linear layers. The adaptive fusion module fuses the input query sample self-attention features with the query sample prediction features at a ratio of λ1, and fuses the support sample self-attention features with the support sample label vectors at a ratio of λ2, thus obtaining the fused features of the query and support samples. and

[0058] 5) Construction of mutual attention networks, such as Figure 4 As shown: Analysis of regional correlations between images based on the fusion features of the support and query samples obtained in step 4).

[0059] 5-1) Input a pair of query sample features that have been fused with attribute features. and supporting sample features First, a dotted convolutional layer is used to transform them into more compact features. and The mutual attention calculation process constructs the correlation tensor based on cosine similarity.

[0060] 5-2) The relevant tensor E is input into a convolutional block h(·) for geometric matching. This convolutional block consists of two four-dimensional convolutional layers, the first of which produces C. l A related tensor, C in this embodiment lThe value is set to 16. The second convolutional layer aggregates these tensors into a single correlation tensor, resulting in the mutual attention correlation tensor D, which represents the matching score between samples. This process achieves geometric matching by analyzing the consistency of adjacent matches in four-dimensional space.

[0061] 5-3) Calculation of the mutual attention map. The mutual attention correlation tensor D is transformed into a query sample mutual attention map Att. q Mutual attention mapping graph Att for supporting samples s The mutual attention map is calculated by converting the matching score between samples into the average matching probability between each position in one sample and all positions in another sample. Multiplying each position of the fused feature by the mutual attention map yields the final feature. The formula for calculating the final feature of a query sample is as follows:

[0062]

[0063] Where, q∈R C To query the final features of the sample, x q Att represents the spatial location on the feature map of the queried sample. q ∈R H×W To query the mutual attention mapping graph of the samples, To query the fused features that combine visual features and attribute information of a sample, ∑ represents the summation operator.

[0064] The formula for calculating the final features of the supporting samples is as follows:

[0065]

[0066] Where, s∈R C To support the final features of the sample, x s Att represents the spatial location on the feature map of the supporting sample. s ∈R H×W To support the mutual attention mapping of samples, To support fusion features that combine visual features and attribute information of samples, ∑ represents the summation operator.

[0067] 6) Based on steps 2), 3), 4), and 5), the convolutional neural network is trained. The training batch size is 16, and the training lasts for 80 epochs. The supervised training method uses the constructed loss function to iteratively optimize the network parameters. After training is complete, the trained model is tested on the test set to obtain the corresponding image classification results.

[0068] 6-1) Compute L using an additional fully connected classification layer on the base network ResNet-12. anchor The loss guides the model classification, and the calculation formula is as follows:

[0069]

[0070] Z a This represents the primary visual features of the query sample, where W and b are the weights and biases in the fully connected layer, respectively, and c∈C. train Indicates the training set category.

[0071] 6-2) For the network framework, including self-attention networks, attribute prediction networks, adaptive fusion networks, and mutual attention networks, end-to-end supervised training is performed as a whole. The metric-based loss-guided model maps query features to prototypes of similar classes, and calculates the metric-based loss L using the cosine similarity between the query sample and the support set prototypes. metric :

[0072]

[0073] Where sim(,) is the cosine similarity, and τ is the scalar coefficient, which is 0.2 in this embodiment. This represents the average of the feature vectors of K query samples from n classes. Similarly, This is a set of prototypes, which is the average of the feature vectors of K support samples from n classes. In this embodiment, K takes the values ​​1 and 5.

[0074] 6-3) For the attribute prediction network, a weighted binary cross-entropy loss function is used to supervise the attribute information as labels.

[0075]

[0076] Where M is the number of attributes, a m It is the tag with the m-th attribute. σ is the predicted value of the m-th attribute, and σ represents the sigmoid activation function.

[0077] 6-4) Finally, combining the definitions of all loss functions and scaling parameters, the following is a summary:

[0078] L = L anchor +vL metric +μL attribute

[0079] Where v and μ are hyperparameters of the balancing loss term. v is set to 2 and μ is set to 1.

[0080] 7) During the testing phase, for each few-shot task, the visual features of all samples are first input into a self-attention network to obtain self-attention features, and the query samples are input into an attribute prediction network to obtain attribute prediction features. The query sample self-attention features and query sample prediction features are adaptively fused, and the support sample self-attention features and support sample label vectors are adaptively fused to obtain corresponding fused features. These fused features are then input into a mutual attention network to obtain the final support and query sample features. Finally, a metric learning method is used to calculate the similarity score between the query sample set and the prototype of each category, and the category with the highest similarity score is used as the predicted label for that query sample. In few-shot image classification tasks, the method of this invention achieves good classification performance. Taking the CUB dataset as an example, in the 5-way 1-shot and 5-way 5-shot tasks, 15 query samples were tested for each class in one task, and the average classification accuracy of the 95% confidence interval of 2000 randomly sampled tasks was output. The accuracy of the method of this invention can reach 78.58% and 90.14%, respectively. Compared to the baseline method, the results are improved by 11.27% and 10.12% respectively. Unlike the method described in this invention, the baseline model does not include self-attention, attribute prediction, adaptive fusion, or mutual attention networks. The classification accuracy demonstrates the effectiveness of the method presented in this invention.

Claims

1. An attribute and attention-guided based few-shot image classification method, characterized in that, The method comprises the following steps: 1) preparing initial data: the initial data comprises image data, category labels and attribute labels; 2) constructing a self-attention network: inputting an image sample into a ResNet-12 network to obtain primary visual features, and then converting the primary visual features of the sample into self-attention features through an attention mechanism to provide reliable input for the next module; 3) constructing an attribute prediction network: inputting a query sample image into a channel attention network to obtain attention features, and then predicting the attributes contained in the current sample through an attribute positioning module; inputting the image features into a channel attention network to obtain attention features, and then inputting the attention features into an attribute positioning module, wherein the attribute positioning module obtains an estimated transformation matrix corresponding to the attention features through a full connection layer; performing bilinear interpolation sampling on the attention features according to the estimated transformation matrix, and then obtaining the final attribute prediction features through a full connection layer; 4) constructing an adaptive fusion network: adaptively fusing the self-attention features output in step 2) and the attribute information in step 3) to obtain corrected image features; Input the query sample attribute prediction feature and the support sample attribute label vector into the weight calculation network to obtain their respective fusion proportion parameters and Wherein the weight calculation network is composed of two linear layers; the adaptive fusion network fuses the input query sample self-attention feature and the query sample prediction feature in proportion The support sample self-attention feature and the support sample label vector are fused in proportion to obtain the fusion feature combining visual features and attribute information. 5) constructing a mutual attention network: analyzing the regional correlation between images according to the fused features of the support sample and the query sample obtained in step 4); 5-1) inputting the fused features of a pair of support sample and query sample; firstly, a point convolution layer is used to convert them into more compact features, and a mutual attention calculation process is used to construct a correlation tensor according to cosine similarity calculation; 5-2) inputting the correlation tensor into a convolution block for geometric matching; the convolution block is composed of two four-dimensional convolution layers, the first convolution layer generates multiple correlation tensors, and the second convolution layer aggregates these tensors into a single correlation tensor, thereby obtaining a mutual attention correlation tensor, which represents the matching score between samples; this process realizes the function of geometric matching by analyzing the consistency of adjacent matches in a four-dimensional space; 5-3) mutual attention mapping calculation; the mutual attention correlation tensor is converted into a mutual attention mapping, and the calculation of the mutual attention mapping is to convert the matching score between samples into the average matching probability of each position in a sample and all positions in another sample; each position of the fused features is multiplied by the mutual attention mapping to obtain the final features; the calculation formula of the final features of the query sample is as follows: ； wherein, is the final feature of the query sample, denotes the spatial position on the feature map of the query sample, is the cross-attention map of the query sample, is the fusion feature of the query sample combining visual features and attribute information, and ∑ denotes a summation operator; the calculation formula of the final features of the support sample is as follows: ； wherein, to support the final features of the sample, represents the spatial position on the feature map of the support sample, is the mutual attention map of the support sample, is the fusion feature of the support sample visual feature combined with the attribute information, and ∑ represents the summation operator; 6) training a convolutional neural network based on steps 2), 3), 4) and 5); the loss function is constructed to iteratively optimize the network parameters; when the training is completed, the trained model is used to test on the test set to obtain the corresponding image classification result; 6-1) Compute using an additional fully connected classification layer on top of the base network ResNet-12 ; this loss guides the model classification, computed as follows: ； wherein represents a primary visual feature of the query sample, and are weights and biases in the fully connected layer, respectively, represents the training set class; 6-2) For the network framework, including self-attention network, attribute prediction network, adaptive fusion network and mutual attention network, the whole is supervised training end to end; the loss based on the measurement guides the model to map the query features to the prototype close to the same class, and the loss based on the measurement is calculated by the cosine similarity between the query sample and the prototype of the support set : ； wherein is the cosine similarity, is a scalar coefficient; denotes the average of the n classes K query sample feature vectors; likewise, is a set of prototypes, the average of the n classes K support sample feature vectors; 6-3) For the attribute prediction network part, a weighted binary cross-entropy loss function is additionally adopted with attribute information as the label for supervision; ； where M is the number of attributes, is the label of the mth attribute, is the predicted value of the mth attribute, denotes the sigmoid activation function; 6-4) the final definition of all loss functions and proportion parameters is as follows: ； where and are hyperparameters balancing the loss terms.

Images