Small sample image classification method combining cross attention and sample generation

Abstract

The invention discloses a small sample image classification method combining cross attention and sample generation. The method comprises the following steps: performing preliminary feature extraction on an input image through a residual network; constructing a cross attention-sample generation network model, and training by using the training set; and inputting a to-be-tested picture, and classifying the to-be-tested picture by using the constructed cross attention-sample generation network model. According to the invention, the problem of low classification accuracy of small sample images is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing, and in particular to a small sample image classification method combining cross attention and sample generation. Background Art

[0002] In recent years, deep learning has flourished and has been widely used in fields such as image processing, and has achieved remarkable results. With its advantages such as high precision and rapid adaptation, deep learning has gradually replaced traditional image feature methods and become the standard algorithm for image classification. However, deep learning relies on ultra-large-scale data sets, and big data has become a constraint on its development.

[0003] In practical applications, such as medicine and military, it is difficult to obtain sufficient labeled sample data due to confidentiality and privacy issues. Finding an alternative image classification scheme is an urgent problem to be solved. In contrast, humans can quickly establish new concept cognition through a small number of examples, so it has attracted the attention of more and more researchers. The concept of small sample learning came into being, which aims to enable machine learning systems to efficiently learn and generalize cognitive abilities from a small number of samples. Small sample learning solves the needs of small training data and manual learning, improves data utilization, and is suitable for fields such as medical research and development, military image recognition, etc. where it is difficult to obtain a large amount of data. At the same time, it reduces the manpower, material resources and time consumption of manual labeling, and has important research value. Therefore, how to effectively utilize the advantages of small sample learning to achieve high-precision small sample image classification is of great research significance.

[0004] Although the current image classification algorithms based on deep learning have achieved many phased results and have high accuracy in many natural image datasets, they are often unable to meet actual needs when applied to some small sample scenarios. Since the amount of data for small sample image classification is very small, the current solutions mainly consider solving this problem from two approaches: one is to make full use of existing data, namely the attention mechanism; the other is to generate additional data, namely the data expansion method. The attention mechanism can make the network focus on specific areas and reduce the interference of redundant information. The current attention mechanism either only uses self-attention and can only focus on the key areas of a picture, or uses mutual attention to focus on the support set and the query set at the same time, but introduces complex networks or convolutions, resulting in increased computational complexity and time complexity. In addition, too few samples will lead to overfitting. How to find available data, generate reliable samples for model training, and design a reasonable sample generation module is another major problem that needs to be solved.

[0005] Small sample learning aims to enable machine learning systems to solve classification problems using only a small number of labeled samples, and to efficiently learn and promote the ability of humans to learn new knowledge. The difficulty of small sample image classification lies mainly in the small number of samples. Small sample classification tasks are defined as N-Way K-Shot tasks, that is, each task has N categories and K images per category. The classic tasks are 5-Way 1-Shot or 5-Way 5-Shot. It is difficult for the network to learn the differences in images and then classify them with such a small number of samples. Even after multiple trainings, due to the small number of samples, the model often has a deep number of layers or a complex structure. In the new task context, it is impossible to establish a mapping with the pre-built model, resulting in overfitting and reducing the classification accuracy. Meta-learning technology can be applied to this task to solve the model mapping problem well. The model is trained through a large number of scenarios. Unlike traditional machine learning, which maps from images to labels, meta-learning is about how to generate a mapping function for each scenario, which is responsible for mapping images to labels. Most of the current small sample classification algorithms optimize the embedding function in the feature space and classify using the principle that the distance between the same category is close and the distance between different categories is far. Summary of the invention

[0006] The present invention provides a small sample image classification method combining cross attention and sample generation. The present invention solves the problem of low accuracy of small sample image classification, as described below:

[0007] A small sample image classification method combining cross attention and sample generation, the method comprising:

[0008] The input image is subjected to preliminary feature extraction through the residual network;

[0009] Construct a cross-attention-sample generation network model and train it using the training set;

[0010] Input the test picture and use the constructed cross-attention-sample generation network model to classify the test picture.

[0011] The cross-attention-sample generation network model includes: a cross-attention transfer module and a sample generation module;

[0012] The cross-transfer attention module includes: 5 convolutional layers, 3 activation layers, 2 Dropout layers and 2 linear layers; the sample generation module includes: a similarity comparator and a sample generation structure.

[0013] Wherein, the cross-attention transfer module is specifically:

[0014] C1_1 is a convolution layer with a kernel size of 1×1, a step size of 1, an input image size of 640×10×10, and an output

[0015] The image size is 1920×10×10, and the number of channels is 1920;

[0016] SM1_1 is the activation layer, using Softmax as the activation function, the input image size is 8×100×100, and the output image size is 8×100×100;

[0017] D1_1 is the Dropout layer with a parameter of 0.5;

[0018] C1_2 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output size of 640×10×10, and a channel size of 640;

[0019] D1_2 is the Dropout layer with a parameter of 0.5;

[0020] C1_3 is a convolution layer with a convolution kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and a channel number of 80;

[0021] C1_4 is a convolution layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and a channel number of 80;

[0022] C1_5 is a convolution layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 640×10×10, and a channel number of 640;

[0023] SM1_2 is the activation layer, using Softmax as the activation function, the input image size is 10×10, and the output image size is 10×10;

[0024] LN1_1 is a linear layer with an input dimension of 640 and an output dimension of 640;

[0025] SM1_3 is the activation layer, using Relu as the activation function, the input image size is 10×10, and the output image size is 10×10;

[0026] LN1_2 is a linear layer with an input dimension of 640 and an output dimension of 640.

[0027] The sample generation module calculates the F-norm of the new task and the base class through a similarity comparator, selects similar categories to construct a new statistical feature, and generates random samples based on this feature.

[0028] Wherein, the method comprises:

[0029] The features are processed by multi-layer perceptron and layer normalization. The structural features extracted by the cross-transfer attention module are complementary to the basic features extracted by the sample generation module. The two representations are combined to form a residual-like structure to obtain the final feature representation.

[0030] The beneficial effects of the technical solution provided by the present invention are:

[0031] 1. This paper improves the classic convolutional neural network model and proposes a CTA-SG model to achieve image classification in small sample scenarios;

[0032] 2. The present invention extracts the relationship between the support image and the query image through the cross-transfer attention module, so that the network is more inclined to pay attention to the similarity between images in the process of extracting feature maps. At the same time, compared with other methods, CTA significantly reduces the computational complexity;

[0033] 3. The present invention can alleviate the harmful effects of insufficient data by migrating the statistical characteristics of the base class and the sample generation (SG) structure;

[0034] 4. The present invention innovatively combines cross-attention with the sample generation mechanism, which not only makes good use of the existing data (attention mechanism), but also introduces additional data (sample generation module) for model training, which is more suitable for the task scenario of small sample image classification and can greatly improve the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a few-shot image classification method combining cross-attention and sample generation;

[0036] Figure 2 Detailed diagram of the implementation process of the meta-training phase;

[0037] Figure 3 Schematic diagram for calculating attention features;

[0038] Figure 4 Schematic diagram of the process of generating samples.

[0039] Table 1 compares the classification accuracy of several methods on the test data set. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0041] The attention mechanism can filter features to make the network focus on what we need, which is divided into self-attention and mutual attention. The self-attention mechanism is for a picture. By assigning weights to different areas, the network pays more attention to the subject and ignores background interference. Mutual attention is for multiple pictures. In small sample classification, it is for the support set and query set pictures, so that the network pays attention to the common parts of the two and ignores the interference of different subjects.

[0042] The embodiment of the present invention constructs a cross-attention module, taking into account the interactive information of the support set and the query set, making full use of existing data, and reusing the self-attention module to reduce the computational time by changing the model input and improve the adaptability of the network. In addition, another difficulty of this task is the overfitting problem caused by the small number of samples. The embodiment of the present invention uses the statistical characteristics of the base class to expand the data for the new task, obtains additional data by finding similar categories, generates reliable samples for model training, and designs a reasonable sample generation module to reduce the risk of overfitting of the network.

[0043] Example 1

[0044] The embodiment of the present invention provides a small sample image classification method combining attention and sample generation, see Figure 1 , the method comprises the following steps:

[0045] 101: The input image is subjected to preliminary feature extraction by ResNet (residual network);

[0046] 102: Construct a CTA-SG (Cross Attention-Sample Generation) network model and train it using the training set;

[0047] 103: Input the image to be tested and classify it using the constructed CTA-SG network model.

[0048] Among them, the deep learning framework used in the embodiment of the present invention is Pytorch [1] . A CTA-SG network model is constructed based on the Pytorch framework. The network model consists of two parts: the cross-transfer attention module and the sample generation module. The cross-transfer attention module includes: 5 convolutional layers, 3 activation layers, 2 Dropout layers and 2 linear layers, see below for details. The sample generation module consists of three formulas, see below for details.

[0049] C1_1 is a convolution layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 1920×10×10, and a channel number of 1920;

[0050] SM1_1 is the activation layer, using Softmax as the activation function, the input image size is 8×100×100, and the output image size is 8×100×100;

[0051] D1_1 is the Dropout layer with a parameter of 0.5;

[0052] C1_2 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output size of 640×10×10, and a channel size of 640;

[0053] D1_2 is the Dropout layer with a parameter of 0.5;

[0054] C1_3 is a convolution layer with a convolution kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and a channel number of 80;

[0055] C1_4 is a convolution layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and a channel number of 80;

[0056] C1_5 is a convolution layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 640×10×10, and a channel number of 640;

[0057] SM1_2 is the activation layer, using Softmax as the activation function, the input image size is 10×10, and the output image size is 10×10;

[0058] LN1_1 is a linear layer with an input dimension of 640 and an output dimension of 640;

[0059] SM1_3 is the activation layer, using Relu as the activation function, the input image size is 10×10, and the output image size is 10×10;

[0060] LN1_2 is a linear layer with an input dimension of 640 and an output dimension of 640.

[0061] The features extracted by the feature extractor are learned by the network itself and cannot focus on the real region of interest (ROI) well. Through the attention module, the embodiment of the present invention further screens the features, combines global features and interactive features, makes full use of existing data, and enhances the adaptability of the network. It is mainly divided into the following three steps:

[0062] In the first step, a single image is passed through the self-attention module to construct a learnable Q, K, and V matrix to calculate the importance of each part in the entire image, thereby removing redundant information.

[0063] In the second step, the self-attention features of the support set and the query set are input into the cross-transfer attention module. Here, two Q, K, and V matrices for learning are constructed. The Q of the support set and the query set are swapped, and K and V remain unchanged. The cross-transfer attention features of the two are calculated respectively. The output feature of the support set is calculated as follows:

[0064]

[0065] Among them, A is the cross-transfer attention feature, Q is the matrix from the support set, K and V are the matrices from the query set, supt is the support set, qry is the query set, cross is the cross-transfer attention feature, T is the matrix transpose, and d is the dimension.

[0066] Different from self-attention, the cross-attention feature is calculated from two images, and the output feature integrates the interactive information of the support set and the query set, allowing the network to focus on the common parts of the two.

[0067] Sometimes, because the data is too small or too large, data loss occurs after Softmax. In this embodiment of the present invention, an adaptive parameter is designed to prevent data from being filtered. The formula is as follows:

[0068]

[0069] Among them, Q is the matrix from the support set, K is the matrix from the query set, supt is the support set, qry is the query set, ∈ is the adaptive parameter, which is set to 0.01 in the experiment, var is the variance, and d is the dimension.

[0070] In the third step, the features are processed by MLP (Multi-layer Perceptron) and LN (Layer Normalization) layers. The structural features extracted by the attention module of the embodiment of the present invention and the basic feature representation extracted by ResNet are often complementary, so these two representations are combined to form a similar residual structure to obtain the final feature representation.

[0071] In summary, the embodiment of the present invention improves the convolutional neural network. The CTA module enables the support set and query set features to interact, extracts the common parts of the two, and removes redundant semantic information.

[0072] Example 2

[0073] The sample generation module and the CTA-SG network model in Example 1 are further introduced below in conjunction with specific examples. For details, see the following description:

[0074] 201: The purpose of sample generation is to transfer the statistical features of the base class to the new task and generate reliable samples for it. It is mainly divided into the following steps (such as Figure 4):

[0075] (1) Calculate the mean and variance of the base class and support set. Assuming that there are 64 base classes and 5 support classes, we get the means of 64*640 and 5*640, and the variances of 64*640*640 and 5*649*640.

[0076] (2) Similarity comparison. Expand the mean matrix of the base class and the support set to obtain a 64*5*640 matrix, and calculate the similarity between the two using the Frobenius norm:

[0077] F=-||E supt -E base ||

[0078] Among them, F is the Frobenius norm, E is the mean, supt is the support set, and base is the base class.

[0079] For a certain category in the support set, the importance of the 64 base categories can be measured by the normalized inverse of the distance.

[0080] (3) Select categories. Select the m categories that are closest to a category in the support set, that is, the first m categories with the greatest importance.

[0081]

[0082] Among them, I is the selected category set, m is the number of categories, is the mean of the i-th category of the base class, and F is the distance calculated above.

[0083] (4) Integrate the statistical features, integrate the m categories with the support set categories, calculate the average value to obtain a new mean and variance, which is the new statistical feature obtained in the feature space in the embodiment of the present invention.

[0084]

[0085] Among them, E is the mean, V is the variance, m is the number of categories, α i is the normalized inverse of the distance, supt is the support set, I(m) is the set obtained above, σ i is the variance corresponding to I(m).

[0086] (5) Generate new samples. In the feature space, similar categories obey the Gaussian distribution. Using the mean variance obtained in 204, the embodiment of the present invention can arbitrarily sample new samples from the Gaussian distribution. These samples can be added to the task as a supplement, effectively alleviating the problem of insufficient data.

[0087] 202: Similarity calculation.

[0088] At present, the embodiment of the present invention needs to compare two groups of data to calculate the similarity. One group is the expanded support set sample, which is composed of original data and newly generated data. The other group is the query set that needs to be classified. Experiments have shown that for the 5-Way 1-Shot task (i.e., there are 5 types of support sets, 1 picture in each category), the inner product is used as the measurement algorithm for better results. For the 5-Way 5-Shot task (i.e., there are 5 types of support sets, 5 pictures in each category), the Euclidean distance is used for similarity measurement. The formulas for the inner product and the Euclidean distance are as follows:

[0089] Inner product calculation formula

[0090] Euclidean distance calculation formula

[0091] Among them, vector A = (x1, y1), vector B = (x2, y2).

[0092] By similarity calculation, the implementation of the present invention can obtain the similarity score of the query set for each category, obtain the label with the highest score and assign it to the query set, thus completing the classification of a meta-task, averaging the classification accuracy of multiple meta-tasks, and obtaining the final classification accuracy. The classification results are detailed in Example 3.

[0093] 203: Build a CTA-SG network model and use the training set for training;

[0094] 204: Use the constructed CTA-SG model to classify the images in the test set.

[0095] Among them, three internationally public benchmark datasets (MiniImageNet, TieredImageNet and CUB-200-2012) were selected for evaluation.

[0096] In summary, the embodiment of the present invention utilizes the proposed cross-attention module and sample generation module, and the input image passes through a primary feature extractor composed of ResNet to map the image to a feature space; then, the primary features pass through the self-attention and cross-attention modules respectively, and the network focuses on the main features and ignores redundant information by screening the features by importance; then, the cross-attention features are compared with the base class for similarity, and reliable statistical features are selected to generate new samples, which are finally classified through similarity measurement.

[0097] Example 3

[0098] The feasibility of the solutions in Examples 1 and 2 is verified below in conjunction with Table 1, as described below:

[0099] 301: The embodiment of the present invention uses three internationally public benchmark datasets (MiniImageNet, TieredImageNet and CUB-200-2012) for evaluation;

[0100] 302: First, the constructed CTA-SG network model is used to train and test the data set, and a trained model is obtained;

[0101] In the experiment of the embodiment of the present invention, the operating system is Ubuntu 20.04, the GPU model is GTX4090, the memory is 24G, and the pytorch deep learning framework is used.

[0102] 303: Experimental results analysis:

[0103] Different classification methods are used to classify the MiniImageNet, TieredImageNet and CUB-200-2012 datasets, and their accuracy is shown in Table 1.

[0104] It can be seen from Table 1 that the classification accuracy of the CTA-SG model designed based on the embodiment of the present invention is not only significantly better than the original network, but also better than other classification methods based on small samples, indicating that the CTA-SG of the embodiment of the present invention can effectively interact with the support set and query set features, generate more samples, and effectively improve network performance, proving the effectiveness of the model.

[0105] Table 1 Comparison of classification accuracy (%) of several methods on different data sets

[0106]

[0107] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A small sample image classification method combining cross attention and sample generation, characterized in that: The method comprises: The input image is subjected to preliminary feature extraction through the residual network; Construct a cross-attention-sample generation network model and train it using the training set; Input the test picture and use the constructed cross-attention-sample generation network model to classify the test picture.

2. The small sample image classification method combining cross attention and sample generation according to claim 1, characterized in that: The cross attention-sample generation network model includes: a cross transfer attention module and a sample generation module; The cross-transfer attention module includes: 5 convolutional layers, 3 activation layers, 2 Dropout layers and 2 linear layers; The sample generation module includes: a similarity comparator and a sample generation structure.

3. The small sample image classification method combining cross attention and sample generation according to claim 2, characterized in that: The cross-attention transfer module is specifically: C1_1 is a convolution layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 1920×10×10, and a channel number of 1920; SM1_1 is the activation layer, using Softmax as the activation function, the input image size is 8×100×100, and the output image size is 8×100×100; D1_1 is the Dropout layer with a parameter of 0.5; C1_2 is a convolutional layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output size of 640×10×10, and a channel size of 640; D1_2 is the Dropout layer with a parameter of 0.5; C1_3 is a convolution layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and a channel number of 80; C1_4 is a convolution layer with a convolution kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 80×10×10, and a channel number of 80; C1_5 is a convolution layer with a kernel size of 1×1, a stride of 1, an input image size of 640×10×10, an output image size of 640×10×10, and a channel number of 640; SM1_2 is the activation layer, using Softmax as the activation function, the input image size is 10×10, and the output image size is 10×10; LN1_1 is a linear layer with an input dimension of 640 and an output dimension of 640; SM1_3 is the activation layer, using Relu as the activation function, the input image size is 10×10, and the output image size is 10×10; LN1_2 is a linear layer with an input dimension of 640 and an output dimension of 640.

4. The small sample image classification method combining cross attention and sample generation according to claim 2, characterized in that: The sample generation module calculates the F-norm of the new task and the base class through a similarity comparator, selects similar categories to construct a new statistical feature, and generates random samples based on this feature.

5. The small sample image classification method combining cross attention and sample generation according to claim 1, characterized in that: The method comprises: The features are processed by multi-layer perceptron and layer normalization. The structural features extracted by the cross-transfer attention module are complementary to the basic features extracted by the sample generation module. The two representations are combined to form a residual-like structure to obtain the final feature representation.

Images