A small sample image classification method based on local orthogonal feature attention fusion
By fusing global features and local features on their orthogonal components, the problem of insufficient feature distribution representation in small sample image classification is solved, thereby improving the accuracy and discriminative power of classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2023-02-10
- Publication Date
- 2026-05-19
AI Technical Summary
Existing few-sample image classification methods struggle to effectively represent the feature distribution of image categories when training samples are lacking, resulting in inaccurate classification performance. In particular, methods based on global or local features perform poorly when there are large differences in the distribution of data features.
By fusing global features and local features on their orthogonal components, information from local and deep features is extracted using shallow networks. Feature representation is enhanced through multi-scale feature fusion and local-global channel attention fusion, generating features that more closely approximate the distribution of class features.
It improves the accuracy of small sample image classification, effectively enhances the representation ability of image features, and ensures accuracy and discriminability in category feature distribution.
Smart Images
Figure CN116109834B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of few-shot image classification technology, and more specifically, to a few-shot image classification method based on local orthogonal feature attention fusion. Background Technology
[0002] Deep learning has achieved tremendous success in the field of image processing, but training a good image recognition model using deep learning often requires a large number of labeled samples. In real life, fields such as medicine and endangered species often face a lack of image data resources, and data privacy and security issues pose significant challenges in collecting sample data in these fields. However, if only a small number of samples are used to train the model, overfitting often occurs. Therefore, few-shot learning has been proposed to address these problems.
[0003] Few-shot learning aims to learn new knowledge from limited data. One important application area of few-shot learning is few-shot image classification. In few-shot image classification, due to the lack of labeled data for the target class, it is generally necessary to use an additional dataset with a large amount of labeled data to train the model and learn prior knowledge for the specific task. This means that few-shot image classification has a key problem to solve: there is no overlap between the classes in the training dataset and the actual test dataset, and their data feature distributions may differ greatly. To overcome this problem, there are currently two main methods in few-shot image classification: meta-learning-based methods and metric learning-based methods. Among them, metric learning-based methods are a research hotspot in this field.
[0004] The core idea of metric learning-based methods is to train a deep embedded neural network that projects image data into a specific feature space and classifies images based on the similarity between image features. Typical methods include Prototype Networks, Relational Networks, and DN4. Prototype Networks and Relational Networks use image-level global features as image representations, classifying query images based on the similarity of global features between query and support images. DN4 uses local features to measure image similarity. This method first uses the k-nearest neighbor algorithm to select the k most similar local features from the support images to each local feature of the query image, and then uses the sum of the similarities between each local feature of the query image and these k local features as the similarity between the query image and the support images. These methods only consider global or local features of the image, making it difficult to effectively represent the feature distribution of image categories when training samples are lacking, leading to inaccurate classification results.
[0005] Existing technology discloses an image classification method, apparatus, and storage medium that integrates attention and mutual information. In this method, an attention classification network and a clustering network that maximizes mutual information are integrated to form the image classification network, thereby improving the network's performance and achieving desirable characteristics. In this classification network, the attention classification network acts as a filter to extract distinguishing features, while the clustering module serves as a semi-supervised learning guide and utilizes semantic similarity inner classes to improve classification performance. Furthermore, the classification network can be trained on data without any class labels, making it suitable for fine-grained image classification with relatively small datasets and improving classification accuracy. However, this scheme, based on attention and mutual information fusion for image classification, does not consider global features or local orthogonal features. Summary of the Invention
[0006] This invention provides a few-sample image classification method based on local orthogonal feature attention fusion. By fusing global features and local features on its orthogonal components, the representation ability of the network is enhanced, thereby improving the accuracy of image classification.
[0007] The primary objective of this invention is to solve the aforementioned technical problems. The technical solution of this invention is as follows:
[0008] A few-sample image classification method based on local orthogonal feature attention fusion includes the following steps:
[0009] S1: Extract shallow features of the image using a feature extraction network. and deep features ;
[0010] S2: For shallow features Channel weighting is performed, and then the weighted features are input into the multi-scale feature fusion module for multi-scale feature extraction and fusion, outputting local features that fuse multi-scale information. ;
[0011] S3: The local features with deep features The input is fed into the local orthogonal feature fusion module, and the output is the fused feature Z;
[0012] S4: Generate a class prototype vector of the support set image or a feature vector of the query set image based on the fusion feature Z;
[0013] S5: Calculate the similarity between the feature vectors of the query set images and the class prototype vectors of each support set image, and output the classification result and classification loss.
[0014] Furthermore, ResNet12 is used as the feature extraction network, which includes four residual blocks. Each residual block contains three sets of 3×3 convolutional layers, batch normalization, and ReLU functions, followed by a max pooling layer; shallow features and deep features These are the features output by the third and fourth residual blocks, respectively.
[0015] Furthermore, channel attention is used for shallow features. Channel weighting is applied, and the channel attention formula is as follows:
[0016]
[0017] in This indicates a global pooling operation. For ReLU operations, For the Sigmoid function, and The number of channels is and 1×1 convolution, These are the features after channel weighting.
[0018] Furthermore, the channel-weighted characteristics are denoted as... We use a set of 1×1 convolutions and k sets of dilated convolutions with different dilation rates to extract features at different scales. The number of channels in each set of convolutions represents the depth features. Channel number For each scale's features, SE channel attention is used to perform local channel weighting. The formula for local channel weighting is:
[0019]
[0020] in For the i-th scale feature, and It consists of two fully connected layers; Channel attention weights generated for each scale feature;
[0021] The softmax function is used globally to adjust the channel weights of feature information at different scales. The adjusted channel weights are then... Represented as:
[0022]
[0023] Each channel feature is weighted using the following formula:
[0024]
[0025] This represents the weighted features of each channel. Finally, the weighted features at different scales are concatenated along the channel dimension, and max pooling is used to reduce the concatenated feature map to a size similar to the deep features. The feature maps are of the same size, resulting in fused multi-scale local features. .
[0026] Furthermore, regarding deep features Global average pooling is used to generate global feature vectors. ; Calculate local features Local feature vectors at each spatial location exist The orthogonal components on the surface are calculated using the following formula:
[0027]
[0028]
[0029] in For global feature vectors, Let (i,j) be the local feature vector of the local feature map at position (i,j). for exist Projection on The calculated local orthogonal eigenvectors; the resulting orthogonal local feature maps Compared with the initial deep features Perform attention fusion and output fused enhanced features.
[0030] Furthermore, the obtained orthogonal local feature maps Compared with the initial deep features Perform an addition operation to obtain the preliminary fused feature X; use the preliminary fused feature to aggregate global and local channel attention, the formula is:
[0031]
[0032] in:
[0033]
[0034]
[0035] in, , , These represent batch normalization, the sigmoid function, and the ReLU activation function, respectively. and The number of channels are respectively and 1×1 convolution; Indicates attention to global channels and local channel attention Perform broadcast summation; apply aggregated global and local channel attention to orthogonal local features. with deep features Attention fusion is performed using the following formula:
[0036]
[0037] in This indicates an element-wise multiplication between two feature maps. This is the final generated fused feature map.
[0038] Furthermore, the class prototype vectors of the support set images or the feature vectors of the query set images include: fused feature maps for the support set images. Generate the corresponding class prototype vector using the following formula:
[0039]
[0040] Where k represents the number of samples in each support set image. When k=1, the class prototype vector of that class is the feature vector generated by the fusion feature map of the corresponding sample.
[0041] For the fused feature map of the query set image The corresponding feature vector is generated directly through global average pooling, as shown in the formula.
[0042] 。
[0043] Furthermore, the cosine similarity between the feature vectors of the query set images and the class prototype vectors of each support set image is calculated. The formula for calculating the cosine similarity is:
[0044]
[0045] in and This represents two vectors with the same dimension. Indicates cosine similarity;
[0046] Furthermore, the classification probability of the query set image is calculated based on cosine similarity, using the following formula:
[0047]
[0048] in Represents the image of the nth query set. The probability of belonging to the k-th category is , It is a temperature coefficient; the final query set image The classification result is the category with the highest probability value, calculated using the following formula:
[0049] .
[0050] Furthermore, the classification loss is calculated based on the classification probability and the true labels of the query set images. The calculation formula is as follows:
[0051]
[0052] in Represents cross-entropy loss, Let be the true label of the nth query set image.
[0053] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:
[0054] This invention utilizes shallow networks to extract information from local and deep features, fusing local feature information at multiple scales to preserve the detailed information of local features, effectively enhancing the feature representation of the image. Simultaneously, to make features of different categories more discriminative, more representative local features from the orthogonal components of global features are used to enhance the original features of the image. Furthermore, local-global channel attention is used to fuse the initial global features and orthogonal local features, further aggregating the global and local features of the image, ensuring that the final feature representation more closely approximates the feature distribution of the category, thereby effectively improving the accuracy of small-sample image classification. Attached Figure Description
[0055] Figure 1 This is a flowchart of a few-sample image classification method based on local orthogonal feature attention fusion according to an embodiment of the present invention.
[0056] Figure 2 This is a schematic diagram of the multi-scale feature fusion module in an embodiment of the present invention.
[0057] Figure 3 This is a schematic diagram of the local orthogonal feature fusion module in an embodiment of the present invention.
[0058] Figure 4 This is a network structure diagram of the feature extraction network ResNet12 in an embodiment of the present invention. Detailed Implementation
[0059] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0060] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0061] Example 1
[0062] like Figure 1 As shown, a few-sample image classification method based on local orthogonal feature attention fusion includes the following steps:
[0063] S1: Extract shallow features of the image using a feature extraction network. and deep features ;
[0064] It should be noted that, as Figure 4 As shown, this invention uses ResNet12 as the feature extraction network. The feature extraction network includes four residual blocks. Each residual block contains three sets of 3×3 convolutions, batch normalization, and ReLU functions forming a convolutional layer. A 2×2 max pooling layer follows each residual block. The number of channels in the convolutional kernels in the four residual blocks are 64, 160, 320, and 640, respectively, and the parameters of the three sets of 3×3 convolutions in each residual block are the same.
[0065] Shallow features and deep features The features output by the third and fourth residual blocks are respectively, where the shallow features contain more local details, while the deep features contain more semantic global information.
[0066] S2: For shallow features Channel weighting is performed, and then the weighted features are input into the multi-scale feature fusion module for multi-scale feature extraction and fusion, outputting local features that fuse multi-scale information. ;
[0067] It should be noted that, in order to preserve important information in local features, this invention focuses on shallow features. Channel weighting is used to initially filter important channels among local features. The channel attention formula is as follows:
[0068]
[0069] in This indicates a global pooling operation. For ReLU operations, For the Sigmoid function, and The number of channels is and 1×1 convolution, These are the features after channel weighting.
[0070] The features after initial channel weighting are input into the multi-scale feature fusion module to obtain multi-scale information of local features. For example... Figure 2 As shown, the channel-weighted features are denoted as... We use a set of 1×1 convolutions and k sets of dilated convolutions with different dilation rates to extract features at different scales. The number of channels in each set of convolutions represents the depth features. Channel number For each scale's features, SE channel attention is used to perform local channel weighting. The formula for local channel weighting is:
[0071]
[0072] in For the i-th scale feature, and It consists of two fully connected layers; Channel attention weights generated for each scale feature;
[0073] The softmax function is used globally to adjust the channel weights of feature information at different scales. The adjusted channel weights are then... Represented as:
[0074]
[0075] Each channel feature is weighted using the following formula:
[0076]
[0077] This represents the weighted features of each channel. Finally, the weighted features at different scales are concatenated along the channel dimension, and max pooling is used to reduce the concatenated feature map to a size similar to the deep features. The feature maps are of the same size, resulting in fused multi-scale local features. .
[0078] S3: The local features with deep features The input is fed into the local orthogonal feature fusion module, and the output is the fused feature Z;
[0079] It should be noted that, as Figure 3 As shown, the local orthogonal feature fusion module mainly consists of two parts: local orthogonal feature extraction and local orthogonal feature attention fusion. The specific steps for local orthogonal feature extraction are as follows:
[0080] For deep features Global average pooling is used to generate global feature vectors. ; Calculate local features Local feature vectors at each spatial location exist The orthogonal components on the surface are calculated using the following formula:
[0081]
[0082]
[0083] in For global feature vectors, Let (i,j) be the local feature vector of the local feature map at position (i,j). for exist Projection on The calculated local orthogonal eigenvectors; the resulting orthogonal local feature maps Compared with the initial deep features Perform attention fusion and output fused enhanced features.
[0084] Orthogonal local feature maps Attention fusion with deep features, the specific steps of orthogonal feature attention fusion are as follows: The obtained orthogonal local feature maps... Compared with the initial deep features Perform an addition operation to obtain the preliminary fused feature X; use the preliminary fused feature to aggregate global and local channel attention, the formula is:
[0085]
[0086] in:
[0087]
[0088]
[0089] in, , , These represent batch normalization, the sigmoid function, and the ReLU activation function, respectively. and The number of channels are respectively and 1×1 convolution; Indicates attention to global channels and local channel attention Perform broadcast summation; apply aggregated global and local channel attention to orthogonal local features. with deep features Attention fusion is performed using the following formula:
[0090]
[0091] in This indicates an element-wise multiplication between two feature maps. This is the final generated fused feature map.
[0092] Example 2
[0093] Based on the above steps, this embodiment elaborates in detail the specific process of generating class prototype vectors of support set images or feature vectors of query set images according to the fusion feature Z.
[0094] S4: Generate a class prototype vector of the support set image or a feature vector of the query set image based on the fusion feature Z;
[0095] It should be noted that the class prototype vector of the support set image or the feature vector of the query set image includes: the fused feature map for the support set image. Generate the corresponding class prototype vector using the following formula:
[0096]
[0097] Where k represents the number of samples in each support set image. When k=1, the class prototype vector of that class is the feature vector generated by the fusion feature map of the corresponding sample.
[0098] For the fused feature map of the query set image The corresponding feature vector is generated directly through global average pooling, as shown in the formula.
[0099] .
[0100] Example 3
[0101] Based on the above steps, this embodiment details the specific steps for calculating the similarity between the feature vector of the query set image and the class prototype vector of each support set image, and outputting the classification result and classification loss.
[0102] S5: Calculate the similarity between the feature vectors of the query set images and the class prototype vectors of each support set image, and output the classification result and classification loss.
[0103] It should be noted that the cosine similarity between the feature vectors of the query set images and the class prototype vectors of each support set image is calculated using the following formula:
[0104]
[0105] in and This represents two vectors with the same dimension. Indicates cosine similarity;
[0106] The classification probability of the query set images is calculated based on cosine similarity, using the following formula:
[0107]
[0108] in Represents the image of the nth query set. The probability of belonging to the k-th category is , It is a temperature coefficient; the final query set image The classification result is the category with the highest probability value, calculated using the following formula:
[0109] .
[0110] The classification loss is calculated based on the classification probability and the true labels of the query set images. The calculation formula is as follows:
[0111]
[0112] in Represents cross-entropy loss, Let be the true label of the nth query set image.
[0113] This invention addresses the problem that existing few-sample image classification methods lack full utilization of global and local features, resulting in generated features that cannot effectively represent the feature distribution of image categories. It proposes a few-sample image classification method based on local orthogonal feature attention fusion. This invention utilizes shallow networks to extract information from local and deep features, fusing local feature information at multiple scales to preserve the detail of local features, effectively enhancing the image's feature representation. Simultaneously, to make features of different categories more discriminative, it utilizes more representative local features from the orthogonal components of global features to enhance the original image features. Furthermore, it employs local-global channel attention to fuse the initial global features and orthogonal local features, further aggregating the global and local features of the image, ensuring that the final generated feature representation more closely approximates the feature distribution of the category, thereby effectively improving the accuracy of few-sample image classification.
[0114] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A few-sample image classification method based on local orthogonal feature attention fusion, characterized in that, Includes the following steps: S1: Extract shallow features of the image using a feature extraction network. and deep features ; S2: For shallow features Channel weighting is performed, and then the weighted features are input into the multi-scale feature fusion module for multi-scale feature extraction and fusion, outputting local features that fuse multi-scale information. ; The channel-weighted features are denoted as follows: We use a set of 1×1 convolutions and k sets of dilated convolutions with different dilation rates to extract features at different scales. The number of channels in each set of convolutions represents the depth features. Channel number For each scale's features, SE channel attention is used to perform local channel weighting. The formula for local channel weighting is: in For the i-th scale feature, and It consists of two fully connected layers; Channel attention weights generated for each scale feature; The softmax function is used globally to adjust the channel weights of feature information at different scales. The adjusted channel weights are then... Represented as: Each channel feature is weighted using the following formula: This represents the weighted features of each channel. Finally, the weighted features at different scales are concatenated along the channel dimension, and max pooling is used to reduce the concatenated feature map to a size similar to the deep features. The feature maps are of the same size, resulting in fused multi-scale local features. ; S3: The local features with deep features The input is fed into the local orthogonal feature fusion module, and the output is the fused feature Z; S4: Generate a class prototype vector of the support set image or a feature vector of the query set image based on the fusion feature Z; S5: Calculate the similarity between the feature vectors of the query set images and the class prototype vectors of each support set image, and output the classification result and classification loss.
2. The few-sample image classification method based on local orthogonal feature attention fusion according to claim 1, characterized in that, ResNet12 is used as the feature extraction network, which includes four residual blocks. Each residual block contains three sets of 3×3 convolutional layers, batch normalization, and ReLU functions. A max pooling layer follows each residual block. Shallow features... and deep features These are the features output by the third and fourth residual blocks, respectively.
3. The few-sample image classification method based on local orthogonal feature attention fusion according to claim 1, characterized in that, Using channel attention for shallow features Channel weighting is applied, and the channel attention formula is as follows: in This indicates a global pooling operation. For ReLU operations, For the Sigmoid function, and The number of channels is and 1×1 convolution, These are the features after channel weighting.
4. The few-sample image classification method based on local orthogonal feature attention fusion according to claim 1, characterized in that, For deep features Global average pooling is used to generate global feature vectors. ; Calculate local features Local feature vectors at each spatial location exist The orthogonal components on the x-axis are calculated using the following formula: in For global feature vectors, Let (i,j) be the local feature vector of the local feature map at position (i,j). for exist Projection on The calculated local orthogonal eigenvectors; the resulting orthogonal local feature maps Compared with the initial deep features Perform attention fusion and output fused enhanced features.
5. A few-sample image classification method based on local orthogonal feature attention fusion according to claim 4, characterized in that, The obtained orthogonal local feature map Compared with the initial deep features Perform an addition operation to obtain the preliminary fused feature X; use the preliminary fused feature to aggregate global and local channel attention, the formula is: in: in, , , These represent batch normalization, the sigmoid function, and the ReLU activation function, respectively. and The number of channels are respectively and 1×1 convolution; Indicates attention to global channels and local channel attention Perform broadcast summation; apply aggregated global and local channel attention to orthogonal local features. with deep features Attention fusion is performed using the following formula: in This indicates an element-wise multiplication between two feature maps. This is the final generated fused feature map.
6. The few-sample image classification method based on local orthogonal feature attention fusion according to claim 1, characterized in that, The class prototype vectors of the support set images or the feature vectors of the query set images include: fused feature maps for the support set images. Generate the corresponding class prototype vector; For the fused feature map of the query set image The corresponding feature vector is generated directly through global average pooling, as shown in the formula: 。 7. The few-sample image classification method based on local orthogonal feature attention fusion according to claim 1, characterized in that, Calculate the cosine similarity between the feature vectors of the query set images and the class prototype vectors of each support set image. The formula for calculating the cosine similarity is: in and This represents two vectors with the same dimension. This represents the cosine similarity.
8. The few-sample image classification method based on local orthogonal feature attention fusion according to claim 7, characterized in that, The classification probability of the query set images is calculated based on cosine similarity, using the following formula: in, Represents the image of the nth query set. The probability of belonging to the k-th category is , It is a temperature coefficient; the final query set image The classification result is the category with the highest probability value, calculated using the following formula: 。 9. A few-sample image classification method based on local orthogonal feature attention fusion according to claim 8, characterized in that, The classification loss is calculated based on the classification probability and the true labels of the query set images. The calculation formula is as follows: in Represents cross-entropy loss, Let be the true label of the nth query set image.