A small sample image classification method combining cross attention and sample generation

Abstract

The application discloses a small sample image classification method combining cross attention and sample generation, and the method comprises the following steps: inputting an image to perform preliminary feature extraction through a residual network; constructing a cross attention-sample generation network model and training the model by using a training set; and inputting a to-be-detected picture and classifying the to-be-detected picture by using the constructed cross attention-sample generation network model. The application solves the problem of low small sample image classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer image processing, and more particularly to a few-sample image classification method that combines cross-attention and sample generation. Background Technology

[0002] In recent years, deep learning has flourished and has been widely applied in fields such as image processing, achieving remarkable results. With its advantages of high accuracy and rapid adaptation, deep learning has gradually replaced traditional image feature methods, becoming the standard algorithm for image classification. However, deep learning relies on extremely large datasets, and big data has become a limiting factor in its development.

[0003] In practical applications, such as medicine and the military, sufficient labeled sample data is difficult to obtain due to confidentiality and privacy concerns. Finding an alternative image classification scheme is a pressing issue. Conversely, humans can quickly develop new conceptual understanding from a small number of examples, thus attracting increasing attention from researchers. The concept of few-shot learning has emerged to address this need, aiming to enable machine learning systems to efficiently learn and generalize cognitive abilities from a limited number of samples. Few-shot learning solves the problems of limited training data and the need for manual learning, improving data utilization. It is suitable for fields such as pharmaceutical research and development and military image recognition where large amounts of data are difficult to obtain, while also reducing the manpower, resources, and time consumed by manual labeling, making it of significant research value. Therefore, how to effectively utilize the advantages of few-shot learning to achieve high-accuracy few-shot image classification is of great research significance.

[0004] While current deep learning-based image classification algorithms have achieved significant progress and high accuracy on many natural image datasets, they often fall short of practical requirements for small-sample scenarios. Due to the limited amount of data available for small-sample image classification, current solutions primarily address this issue through two approaches: fully utilizing existing data (i.e., attention mechanisms) and generating additional data (i.e., data augmentation methods). Attention mechanisms allow the network to focus on specific regions, reducing redundant information interference. Current attention mechanisms either use only self-attention, focusing only on key regions of an image, or use mutual attention to simultaneously focus on the support and query sets, but this introduces complex networks or convolutions, increasing computational and time complexity. Furthermore, insufficient samples can lead to overfitting. Therefore, finding usable data, generating reliable samples for model training, and designing a reasonable sample generation module are other major problems that urgently need to be solved.

[0005] Few-shot learning aims to enable machine learning systems to solve classification problems using only a small number of labeled samples, efficiently learning and generalizing information similar to how humans learn new knowledge. The difficulty of few-shot image classification lies primarily in the limited number of samples. Few-shot classification tasks are defined as N-Way K-Shot tasks, meaning each task has N classes and K images per class. Classic tasks include 5-Way 1-Shot or 5-Way 5-Shot. Networks struggle to learn the differences between images and classify them using such a small number of samples. Even after multiple training iterations, due to the limited sample size, models often have deep layers or complex structures, making it difficult to establish a mapping with the pre-built model in new task contexts, leading to overfitting and reduced classification accuracy. Meta-learning techniques can effectively solve the model mapping problem in this task. The model is trained through a large number of scenarios. Unlike traditional machine learning, which maps images to labels, meta-learning generates a mapping function for each scenario, responsible for mapping images to labels. Most current few-shot classification algorithms optimize embedding functions in the feature space, using the principle of proximity between similar classes and distance between dissimilar classes for classification. Summary of the Invention

[0006] This invention provides a few-sample image classification method that combines cross-attention and sample generation. This invention solves the problem of low classification accuracy for few-sample images, as detailed below:

[0007] A few-sample image classification method combining cross-attention and sample generation, the method comprising:

[0008] The input image undergoes preliminary feature extraction via a residual network;

[0009] Construct a cross-attention-sample generation network model and train it using the training set;

[0010] Input the image to be tested, and use the constructed cross-attention-sample generation network model to classify the image.

[0011] The cross-attention-sample generation network model includes: a cross-transfer attention module and a sample generation module;

[0012] The cross-transfer attention module includes: 5 convolutional layers, 3 activation layers, 2 dropout layers, and 2 linear layers; the sample generation module includes: a similarity comparator and a sample generation structure.

[0013] Specifically, the cross-transfer attention module is as follows:

[0014] C1_1 is a convolutional layer with a kernel size of 1×1 and a stride of 1. The input image size is 640×10×10, and the output...

[0015] The image size is 1920×10×10, and the number of channels is 1920.

[0016] SM1_1 is the activation layer, using Softmax as the activation function. The input image size is 8×100×100, and the output image size is 8×100×100.

[0017] D1_1 is the Dropout layer with a parameter of 0.5;

[0018] C1_2 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 640×10×10, and a channel size of 640.

[0019] D1_2 is the Dropout layer with a parameter of 0.5;

[0020] C1_3 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and 80 channels.

[0021] C1_4 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and 80 channels.

[0022] C1_5 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 640×10×10, and 640 channels.

[0023] SM1_2 is the activation layer, using Softmax as the activation function. The input image size is 10×10, and the output image size is 10×10.

[0024] LN1_1 is a linear layer with an input dimension of 640 and an output dimension of 640.

[0025] SM1_3 is the activation layer, using ReLU as the activation function. The input image size is 10×10, and the output image size is 10×10.

[0026] LN1_2 is a linear layer with an input dimension of 640 and an output dimension of 640.

[0027] The sample generation module calculates the F-norm of the new task and the base class using a similarity comparator, selects similar categories to construct new statistical features, and generates random samples based on these features.

[0028] The method includes:

[0029] The features are processed by multilayer perceptron and layer normalization. The structural features extracted by the cross-transfer attention module are complementary to the basic features extracted by the sample generation module. Combining these two representations forms a residual-like structure, resulting in the final feature representation.

[0030] The beneficial effects of the technical solution provided by this invention are:

[0031] 1. This invention improves upon the classic convolutional neural network model and proposes a CTA-SG model, which realizes image classification in scenarios with few samples;

[0032] 2. This invention extracts the relationship between support images and query images through the cross-transfer attention module, making the network more inclined to focus on the similarity between images during the feature map extraction process. At the same time, compared with other methods, CTA significantly reduces the computational complexity.

[0033] 3. By transferring the statistical characteristics of the base class, the sample generation (SG) structure of this invention can mitigate the harmful effects of insufficient data;

[0034] 4. This invention innovatively combines cross-attention with a sample generation mechanism, which not only makes good use of existing data (attention mechanism) but also introduces additional data (sample generation module) to supply model training, making it more suitable for small sample image classification tasks and significantly improving the accuracy of image classification. Attached Figure Description

[0035] Figure 1 A flowchart of a few-sample image classification method that combines cross-attention and sample generation;

[0036] Figure 2 A detailed diagram illustrating the implementation process of the meta-training phase;

[0037] Figure 3 A schematic diagram for calculating attention features;

[0038] Figure 4 This is a schematic diagram of the sample generation process.

[0039] Table 1 compares the classification accuracy of several methods on the test dataset. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0041] Attention mechanisms, used for feature selection, allow the network to focus on the areas we need. These mechanisms are categorized into self-attention and mutual attention. Self-attention works on a single image, assigning weights to different regions to make the network focus more on the subject and ignore background interference. Mutual attention works on multiple images; in few-shot classification, it applies to the support set and query set images, causing the network to focus on the common parts of both sets and ignore interference from different subjects.

[0042] This invention constructs a cross-attention module that considers the interaction information between the support set and the query set, making full use of existing data. It also reuses the self-attention module, reducing computation time by changing the model input and improving the network's adaptability. Furthermore, another challenge of this task is overfitting due to insufficient sample size. This invention uses statistical features of the base classes to augment the data for the new task, obtaining additional data by finding similar categories, generating reliable samples for model training, and designing a reasonable sample generation module to reduce the risk of overfitting.

[0043] Example 1

[0044] This invention provides a few-sample image classification method that combines attention and sample generation. See [link to relevant documentation]. Figure 1 The method includes the following steps:

[0045] 101: The input image undergoes preliminary feature extraction using a ResNet (residual network);

[0046] 102: Construct a CTA-SG (Cross-Attention-Sample Generation) network model and train it using the training set;

[0047] 103: Input the image to be tested and classify it using the constructed CTA-SG network model.

[0048] The deep learning framework used in this embodiment of the invention is PyTorch. [1] A CTA-SG network model was constructed based on the PyTorch framework. This network model consists of two parts: a cross-transfer attention module and a sample generation module. The cross-transfer attention module includes 5 convolutional layers, 3 activation layers, 2 dropout layers, and 2 linear layers, as detailed below. The sample generation module consists of three formulas, as detailed below.

[0049] C1_1 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 1920×10×10, and 1920 channels.

[0050] SM1_1 is the activation layer, using Softmax as the activation function. The input image size is 8×100×100, and the output image size is 8×100×100.

[0051] D1_1 is the Dropout layer with a parameter of 0.5;

[0052] C1_2 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 640×10×10, and 640 channels.

[0053] D1_2 is the Dropout layer with a parameter of 0.5;

[0054] C1_3 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and 80 channels.

[0055] C1_4 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and 80 channels.

[0056] C1_5 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 640×10×10, and 640 channels.

[0057] SM1_2 is the activation layer, using Softmax as the activation function. The input image size is 10×10, and the output image size is 10×10.

[0058] LN1_1 is a linear layer with an input dimension of 640 and an output dimension of 640.

[0059] SM1_3 is the activation layer, using ReLU as the activation function. The input image size is 10×10, and the output image size is 10×10.

[0060] LN1_2 is a linear layer with an input dimension of 640 and an output dimension of 640.

[0061] The features extracted by the feature extractor are learned autonomously by the network and cannot effectively focus on the true Region of Interest (ROI). Through the attention module, this embodiment of the invention further filters the features, combining global and interactive features to fully utilize existing data and enhance network adaptability. This mainly consists of the following three steps:

[0062] The first step is to pass a single image through a self-attention module. By constructing a learnable Q, K, V matrix, the importance of each part in the whole image is calculated, thereby removing redundant information.

[0063] The second step involves inputting the self-attention features of the support set and query set into the cross-transfer attention module. Here, two learnable Q, K, and V matrices are constructed. The Q matrices of the support set and query set are swapped, while K and V remain unchanged. The cross-transfer attention features of the two matrices are then calculated separately. The output features of the support set are calculated as follows:

[0064]

[0065] Where A is the cross-transfer attention feature, Q is a matrix from the support set, K and V are matrices from the query set, supt is the support set, qry is the query set, cross is the cross-transfer attention feature, T is the matrix transpose, and d is the dimension.

[0066] Unlike self-attention, cross-transfer attention features are computed from two images, and the output features integrate the interaction information of the support set and the query set, enabling the network to focus on the common parts of the two.

[0067] Sometimes, due to data being too small or too large, data loss occurs after passing through Softmax. This invention addresses this by designing an adaptive parameter to prevent data filtering, as shown in the following formula:

[0068]

[0069] Where Q is a matrix from the support set, K is a matrix from the query set, supt is the support set, qry is the query set, ∈ is the adaptive parameter (set to 0.01 in the experiment), var is the variance, and d is the dimension.

[0070] The third step involves processing the features using MLP (Multilayer Perceptron) and LN (Layer Normalization) layers. The structural features extracted by the attention module in this embodiment are often complementary to the basic feature representations extracted by ResNet. Therefore, combining these two representations forms a residual-like structure, yielding the final feature representation.

[0071] In summary, the embodiments of the present invention have improved the convolutional neural network. The CTA module enables the interaction of support set and query set features, extracts the common parts between the two, and removes redundant semantic information.

[0072] Example 2

[0073] The sample generation module in Example 1 and the CTA-SG network model will be further described below with specific examples:

[0074] 201: The purpose of sample generation is to transfer the statistical features of the base class to the new task and generate reliable samples for it. It mainly consists of the following steps (e.g., Figure 4):

[0075] (1) Calculate the mean and variance of the base class and the support set. Assume there are 64 base classes and 5 support sets. The mean of 64*640 and 5*640 is obtained, and the variance of 64*640*640 and 5*649*640 is obtained.

[0076] (2) Similarity comparison. The mean matrices of the base class and support sets are expanded to obtain a 64*5*640 matrix. The similarity between the two is calculated using the Frobenius norm:

[0077] F = -||E supt -E base ||

[0078] Where F is the Frobenius norm, E is the mean, supt is the support set, and base is the base class.

[0079] For a given class in the support set, the importance of the 64 base classes can be measured by the normalized inverse of the distance.

[0080] (3) Select categories: Select the m categories that are most similar to a certain category in the support set, that is, the top m categories with the highest importance.

[0081]

[0082] Where I is the selected set of categories, and m is the number of categories. Let F be the mean of the i-th category of the base class, and F be the distance calculated above.

[0083] (4) Integrate statistical features, integrate the m classes with the support set classes, calculate the average value to obtain the new mean and variance, which is the new statistical feature obtained in the feature space in the embodiment of the present invention.

[0084]

[0085] Where E is the mean, V is the variance, m is the number of categories, and α is the variance. i Let σ be the normalized reciprocal of the distance, supt be the support set, I(m) be the set obtained above, and σ be the normalized reciprocal of the distance. i Let I(m) be the variance.

[0086] (5) Generate new samples. In the feature space, similar categories follow a Gaussian distribution. Using the mean and variance obtained by 204, this embodiment of the invention can arbitrarily sample new samples from the Gaussian distribution. These samples can be added to the task as a supplement, effectively alleviating the problem of insufficient data.

[0087] 202: Similarity calculation.

[0088] In this embodiment of the invention, two sets of data are compared to calculate similarity. One set is the expanded support set samples, which consists of the original data and newly generated data. The other set is the query set to be classified. Experiments show that for the 5-Way 1-Shot task (i.e., there are 5 support sets, with 1 image per set), using the inner product as the metric algorithm is more effective. For the 5-Way 5-Shot task (i.e., there are 5 support sets, with 5 images per set), using Euclidean distance for similarity measurement is more effective. The formulas for the inner product and Euclidean distance are as follows:

[0089] Inner product calculation formula

[0090] Euclidean distance calculation formula

[0091] Where vector A = (x1, y1) and vector B = (x2, y2).

[0092] By calculating the similarity score, the similarity score of the query set for each class can be obtained. The label with the highest score is assigned to the query set, thus completing the classification of one meta-task. The classification accuracy of multiple meta-tasks is averaged to obtain the final classification accuracy. The classification results are detailed in Example 3.

[0093] 203: Construct the CTA-SG network model and train it using the training set;

[0094] 204: Use the constructed CTA-SG model to classify the images in the test set.

[0095] Three internationally available benchmark datasets (MiniImageNet, TieredImageNet, and CUB-200-2012) were selected for evaluation.

[0096] In summary, this embodiment of the invention utilizes the proposed cross-transfer attention module and sample generation module. The input image is mapped to the feature space by a primary feature extractor composed of ResNet. Then, the primary features are processed by self-attention and cross-attention modules, which filter the importance of features so that the network focuses on the main features and ignores redundant information. Next, the cross-attention features are compared with the base class to select reliable statistical features to generate new samples. Finally, classification is performed using a similarity metric.

[0097] Example 3

[0098] The feasibility of the schemes in Examples 1 and 2 is verified below with reference to Table 1, as detailed in the following description:

[0099] 301: This embodiment of the invention uses three internationally published benchmark datasets (MiniImageNet, TieredImageNet, and CUB-200-2012) for evaluation;

[0100] 302: First, the dataset is trained and tested using the constructed CTA-SG network model to obtain the trained model;

[0101] In the experiments of this embodiment of the invention, the operating system was Ubuntu 20.04, the GPU model was GTX4090, the memory was 24G, and the PyTorch deep learning framework was used.

[0102] 303: Analysis of Experimental Results:

[0103] The MiniImageNet, TieredImageNet, and CUB-200-2012 datasets were classified using different classification methods, and their accuracy is shown in Table 1.

[0104] As can be seen from Table 1, the CTA-SG model designed based on the embodiments of the present invention not only has a significantly better classification accuracy than the original network, but also outperforms other classification methods based on small samples. This indicates that the CTA-SG model of the present invention can effectively interact with the support set and query set features to generate more samples, which can effectively improve network performance and prove the effectiveness of the model.

[0105] Table 1 Comparison of classification accuracy (%) of several methods on different datasets

[0106]

[0107] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A few-sample image classification method combining cross-attention and sample generation, characterized in that, The method includes: The input image undergoes preliminary feature extraction via a residual network; Construct a cross-attention-sample generation network model and train it using the training set; Input the image to be tested, and use the constructed cross-attention-sample generation network model to classify the image to be tested; The cross-attention-sample generation network model includes: a cross-transfer attention module and a sample generation module; The cross-transfer attention module includes: 5 convolutional layers, 3 activation layers, 2 dropout layers, and 2 linear layers. layer; The sample generation module includes: a similarity comparator and a sample generation structure; The cross-attention transfer module is specifically as follows: C1_1 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 1920×10×10, and 1920 channels. SM1_1 is the activation layer, using Softmax as the activation function. The input image size is 8×100×100, and the output image size is 8×100×100. D1_1 is the Dropout layer with a parameter of 0.5; C1_2 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 640×10×10, and 640 channels. D1_2 is the Dropout layer with a parameter of 0.5; C1_3 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and 80 channels. C1_4 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and 80 channels. C1_5 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 640×10×10, and 640 channels. SM1_2 is the activation layer, using Softmax as the activation function. The input image size is 10×10, and the output image size is 10×10. LN1_1 is a linear layer with an input dimension of 640 and an output dimension of 640. SM1_3 is the activation layer, using ReLU as the activation function. The input image size is 10×10, and the output image size is 10×10. LN1_2 is a linear layer with an input dimension of 640 and an output dimension of 640. The method includes: The features are processed by multilayer perceptron and layer normalization. The structural features extracted by the cross-transfer attention module are complementary to the basic features extracted by the residual network. Combining these two representations forms a residual-like structure, resulting in the final feature representation. The sample generation module calculates the F-norm of the new task and the base class using a similarity comparator, selects similar categories to construct new statistical features, and generates random samples based on these features. Specifically: (1) Calculate the mean and variance of the base class and the support set. Assume there are 64 base classes and 5 support sets. The mean of 64*640 and 5*640 is obtained, and the variance of 64*640*640 and 5*649*640 is obtained. (2) Expand the mean matrix of the base class and support set to obtain a 64*5*640 matrix, and calculate the similarity between them using the Frobenius norm: Where F is the Frobenius norm, E is the mean, supt is the support set, and base is the base class; (3) Select the categories: Select the m categories that are most similar to a certain category in the support set, that is, the top m categories with the highest importance. Don't: Where I is the selected set of categories, and m is the number of categories. Let be the mean of the i-th category of the base class; (4) Integrate statistical features, combine the m classes with the support set classes, and calculate the average to obtain a new mean and variance: Where E is the mean, V is the variance, and m is the number of categories. Let be the normalized reciprocal of the distance, and supt be the support set. For a set, for The corresponding variance; (5) In the feature space, similar categories follow a Gaussian distribution. Using the obtained mean and variance, new samples are randomly sampled from the Gaussian distribution and added to the task as a supplement.

Images